# [[Deploying Loki on GCP GKE via Helm]] ![[Deploying Loki on GCP GKE via Helm.svg]] You can [[How to deploy Grafana Loki|deploy]] [[Grafana Loki|Loki]] on [[Kubernetes]] in [[Google Cloud Platform|GCP]] using: - [[Google Cloud Storage]] as a storage for Loki - [[Google Kubernetes Engine|GKE]] running on [[Google Compute Engine]] worker nodes - [[Helm for Kubernetes|Helm]] charts to handle configuration ## Prerequisites - [[kubectl]] - [[Helm for Kubernetes|Helm]] - [[gcloud CLI]] - A [[Google Cloud Platform|GCP]] account that you can start clusters on ## Create cluster Run this command to create the cluster: ```bash gcloud container clusters create loki-gcp \ --location=<REGION> \ --num-nodes=1 \ --machine-type=<MACHINE_TYPE> \ --release-channel=regular \ --workload-pool=<PROJECT_ID>.svc.id.goog \ --enable-ip-alias \ --no-enable-basic-auth \ --no-issue-client-certificate ``` Fill in the `< >` variables as necessary. For `<MACHINE_TYPE>`, we're using `n2-standard-8` machines, [which have](https://gcloud-compute.com/n2-standard-8.html): - 8 cores - 32 GB RAM - 16 Gbps network bandwidth For `workload-pool`, replace `<PROJECT_ID>` with the GCP project ID (not the numeric project number). For example: `workload-pool: loki-demo.svc.id.goog`. Here's an example of what that looks like with everything filled in: ```bash gcloud container clusters create loki-gcp \ --location=europe-west4 \ --num-nodes=1 \ --machine-type=n2-standard-8 \ --release-channel=regular \ --workload-pool=dev-advocacy-380120.svc.id.goog \ --enable-ip-alias \ --no-enable-basic-auth \ --no-issue-client-certificate ``` The command above will create 3 nodes in the region specified. > [!question]- Why does the command create 3 nodes when `num_nodes=1`? > Regional clusters in GKE are designed for resilience, and thus by default span three zones within the region. If we set `num_nodes=3`, we would have 9 nodes in total for the region: 3 in *each* zone. ## Create [[Google Cloud Storage]] storage buckets ```bash gcloud storage buckets create gs://<CHUNKS_BUCKET_NAME> gs://<RULER_BUCKET_NAME> \ --location=<REGION> \ --default-storage-class=STANDARD \ --public-access-prevention \ --uniform-bucket-level-access \ --soft-delete-duration=7d ``` Replace `<CHUNKS_BUCKET_NAME>` and `<RULER_BUCKET_NAME>` with unique names for the chunks and ruler buckets that are not just `ruler` and `chunks`. Replace `<REGION>` with the correct region. Here's an example with all the values filled in: ```bash gcloud storage buckets create gs://loki-gcp-chunks gs://loki-gcp-ruler \ --location=europe-west4 \ --default-storage-class=STANDARD \ --public-access-prevention \ --uniform-bucket-level-access \ --soft-delete-duration=7d ``` You should get this in response: ```bash Creating gs://loki-gcp-chunks/... Creating gs://loki-gcp-ruler/... ``` ## Set permissions for Loki to access GCS ### Authenticate to the GKE cluster Now we want to be able to run `kubectl` commands on the cluster, so make sure you have it installed (run `gcloud components install kubectl` if not) and then run this command: ```bash gcloud container clusters get-credentials loki-gcp \ --region=europe-west4 ``` This will authenticate you via your GCP IAM identity, write the cluster's access info to your local kubeconfig (usually `~/.kube/config`), and then lets `kubectl` commands talk to the right cluster from now on. Then check that you're connected to the GKE cluster and that you're accessing it via `kubectl` by running: ```bash kubectl config current-context ``` You should get something like this in return: ```bash gke_dev-advocacy-380120_europe-west4_loki-gcp ``` ### Create a Kubernetes Namespace Create a K8s namespace where you'll install your Loki workloads: ```bash kubectl create namespace <NAMESPACE> ``` Replace `<NAMESPACE>` with the namespace where your Loki workloads will be located. Example: ```bash kubectl create namespace loki ``` You should get the output: ```bash namespace/loki created ``` ### Create Kubernetes Service Account (KSA) A KSA is a cluster identity (service account, named `default` by default) assigned to pods that allows pods to interact with each other. Create a KSA on your K8s cluster: ```bash kubectl create serviceaccount <KSA_NAME> \ --namespace <NAMESPACE> ``` Replace `<KSA_NAME>` with the name of the KSA created above, and `<NAMESPACE>` with the namespace where your Loki/GEL workloads are located. Example: ```bash kubectl create serviceaccount loki-gcp-ksa \ --namespace loki ``` You should get this in response: ```bash serviceaccount/loki-gcp-ksa created ``` ### Add IAM Policy to Bucket(s) > [!NOTE] > The [pre-defined `roles/storage.objectUser` role](https://cloud.google.com/storage/docs/access-control/iam-roles) is sufficient for Loki / GEL to operate. See [IAM permissions for Cloud Storage](https://cloud.google.com/storage/docs/access-control/iam-permissions) for details about each individual permission. You can use this predefined role or create your own with matching permissions. Create an IAM policy binding on the bucket(s) using the KSA created previously and the role(s) of your choice. One command per bucket. ```txt gcloud storage buckets add-iam-policy-binding gs://<BUCKET_NAME> \ --role=roles/storage.objectUser \ --member=principal://iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<PROJECT_ID>.svc.id.goog/subject/ns/<NAMESPACE>/sa/<KSA_NAME> \ --condition=None ``` Replace `<PROJECT_ID>` with the GCP project ID (ex. project-name), `<PROJECT_NUMBER>` with the project number (ex. 1234567890), `<NAMESPACE>` with the namespace where Loki/GEL is installed, and `<KSA_NAME>` with the name of the KSA you created above. Example: ```bash gcloud storage buckets add-iam-policy-binding gs://loki-gcp-chunks \ --role=roles/storage.objectUser \ --member=principal://iam.googleapis.com/projects/93209135917/locations/global/workloadIdentityPools/dev-advocacy-380120.svc.id.goog/subject/ns/loki/sa/loki-gcp-ksa \ --condition=None ``` You should get this in response: ```bash bindings: - members: - projectEditor:dev-advocacy-380120 - projectOwner:dev-advocacy-380120 role: roles/storage.legacyBucketOwner - members: - projectViewer:dev-advocacy-380120 role: roles/storage.legacyBucketReader - members: - projectEditor:dev-advocacy-380120 - projectOwner:dev-advocacy-380120 role: roles/storage.legacyObjectOwner - members: - projectViewer:dev-advocacy-380120 role: roles/storage.legacyObjectReader - members: - principal://iam.googleapis.com/projects/93209135917/locations/global/workloadIdentityPools/dev-advocacy-380120.svc.id.goog/subject/ns/loki/sa/loki-gcp-ksa role: roles/storage.objectUser etag: CAI= kind: storage#policy resourceId: projects/_/buckets/loki-gcp-chunks version: 1 ``` ## Add authentication for Loki (optional but recommended) ```bash brew install httpd ``` It's part of the Apache HTTP Server, which includes the `htpasswd` utility. ```bash htpasswd -c .htpasswd loki ``` It will ask you for your password (twice). %% ```bash trickster ``` %% Create a Kubernetes secret with the `.htpasswd` file: ```bash kubectl create secret generic loki-basic-auth --from-file=.htpasswd -n loki ``` It should say: ```bash secret/loki-basic-auth created ``` Create a `canary-basic-auth` secret for the canary: ```bash kubectl create secret generic canary-basic-auth \ --from-literal=username=loki \ --from-literal=password=<LOKI_PASSWORD> \ -n loki ``` It should return: ```bash secret/canary-basic-auth created ``` ## Deploy Loki via Helm ### Set up Helm Add Grafana repo: ```bash helm repo add grafana https://grafana.github.io/helm-charts ``` Update the chart repository: ```bash helm repo update ``` Create a new namespace for Loki: ``` kubectl create namespace loki ``` ### Create Helm config Create a `values.yaml` file: ```yaml loki: schemaConfig: configs: - from: "2024-04-01" store: tsdb object_store: gcs schema: v13 index: prefix: loki_index_ period: 24h storage_config: gcs: bucket_name: loki-gcp-chunks # Your actual gcs bucket name, for example, loki-aws-dev-chunks ingester: chunk_encoding: snappy pattern_ingester: enabled: true limits_config: allow_structured_metadata: true volume_enabled: true retention_period: 672h # 28 days retention compactor: retention_enabled: true delete_request_store: gcs ruler: enable_api: true storage_config: type: gcs gcs_storage_config: region: europe-west4 bucketnames: loki-gcp-ruler # Your actual gcs bucket name, for example, loki-aws-dev-ruler alertmanager_url: http://prom:9093 # The URL of the Alertmanager to send alerts (Prometheus, Mimir, etc.) querier: max_concurrent: 4 storage: type: gcs bucketNames: chunks: loki-gcp-chunks # Your actual gcs bucket name (loki-aws-dev-chunks) ruler: loki-gcp-ruler # Your actual gcs bucket name (loki-aws-dev-ruler) serviceAccount: create: false name: loki-gcp-ksa deploymentMode: Distributed ingester: replicas: 3 zoneAwareReplication: enabled: false querier: replicas: 3 maxUnavailable: 2 queryFrontend: replicas: 2 maxUnavailable: 1 queryScheduler: replicas: 2 distributor: replicas: 3 maxUnavailable: 2 compactor: replicas: 1 indexGateway: replicas: 2 maxUnavailable: 1 ruler: replicas: 1 maxUnavailable: 1 # This exposes the Loki gateway so it can be written to and queried externaly gateway: service: type: LoadBalancer basicAuth: enabled: true existingSecret: loki-basic-auth # Since we are using basic auth, we need to pass the username and password to the canary lokiCanary: extraArgs: - -pass=$(LOKI_PASS) - -user=$(LOKI_USER) extraEnv: - name: LOKI_PASS valueFrom: secretKeyRef: name: canary-basic-auth key: password - name: LOKI_USER valueFrom: secretKeyRef: name: canary-basic-auth key: username # Enable minio for storage minio: enabled: false backend: replicas: 0 read: replicas: 0 write: replicas: 0 singleBinary: replicas: 0 ``` ### Deploy Helm config ```bash helm install --values values.yaml loki grafana/loki -n loki --create-namespace ``` ## Test Loki deployment #### Find Loki Gateway Service ```bash kubectl get svc -n loki ``` Shows: ```bash loki-gateway LoadBalancer 34.118.239.140 34.91.203.240 80:30566/TCP 25m ``` Choose the one in the `EXTERNAL-IP` column: ``` 34.91.203.240 ``` #### Test ##### The Script ```javascript import {sleep, check} from 'k6'; import loki from 'k6/x/loki'; /** * URL used for push and query requests * Path is automatically appended by the client * @constant {string} */ const username = '<USERNAME>'; const password = '<PASSWORD>'; const external_ip = '<EXTERNAL-IP>'; const credentials = `${username}:${password}`; const BASE_URL = `http://${credentials}@${external_ip}`; /** * Helper constant for byte values * @constant {number} */ const KB = 1024; /** * Helper constant for byte values * @constant {number} */ const MB = KB * KB; /** * Instantiate config and Loki client */ const conf = new loki.Config(BASE_URL); const client = new loki.Client(conf); /** * Define test scenario */ export const options = { vus: 10, iterations: 10, }; export default () => { // Push request with 10 streams and uncompressed logs between 800KB and 2MB var res = client.pushParameterized(10, 800 * KB, 2 * MB); // Check for successful write check(res, { 'successful write': (res) => res.status == 204 }); // Pick a random log format from label pool let format = randomChoice(conf.labels["format"]); // Execute instant query with limit 1 res = client.instantQuery(`count_over_time({format="${format}"}[1m])`, 1) // Check for successful read check(res, { 'successful instant query': (res) => res.status == 200 }); // Execute range query over last 5m and limit 1000 res = client.rangeQuery(`{format="${format}"}`, "5m", 1000) // Check for successful read check(res, { 'successful range query': (res) => res.status == 200 }); // Wait before next iteration sleep(1); } /** * Helper function to get random item from array */ function randomChoice(items) { return items[Math.floor(Math.random() * items.length)]; } ``` ##### Running the script Replace external-ip. Used [[g]] [[Package Manager]] to update my version of [[Go]]. Then I can install [[Extensions for k6 using xk6|xk6]]: ```bash go install go.k6.io/xk6/cmd/xk6@latest ``` Then clone xk6 repo: ```bash git clone https://github.com/grafana/xk6-loki cd xk6-loki ``` Then make a new binary of k6: ```bash make k6 ``` Then run the test: ```bash ./k6 run gcp-test.js ``` %% ## To do 1. Read [Jay's guide for](https://grafana.com/docs/loki/latest/setup/install/helm/deployment-guides/aws/) [[Deploying Loki on AWS]]. 2. Swap over to GCP's object storage %% ## Resources - [[Sean Pedersen]]'s notes [here](https://github.com/skpaz/grafana/tree/main/loki/storage) - %% # Excalidraw Data ## Text Elements ## Drawing ```json { "type": "excalidraw", "version": 2, "source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.1.4", "elements": [ { "id": "4y8R7iOA", "type": "text", "x": 118.49495565891266, "y": -333.44393157958984, "width": 3.8599853515625, "height": 24, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "roundness": null, "seed": 967149026, "version": 2, "versionNonce": 939059582, "isDeleted": true, "boundElements": null, "updated": 1713723615080, "link": null, "locked": false, "text": "", "rawText": "", "fontSize": 20, "fontFamily": 4, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "", "lineHeight": 1.2 } ], "appState": { "theme": "dark", "viewBackgroundColor": "#ffffff", "currentItemStrokeColor": "#1e1e1e", "currentItemBackgroundColor": "transparent", "currentItemFillStyle": "solid", "currentItemStrokeWidth": 2, "currentItemStrokeStyle": "solid", "currentItemRoughness": 1, "currentItemOpacity": 100, "currentItemFontFamily": 4, "currentItemFontSize": 20, "currentItemTextAlign": "left", "currentItemStartArrowhead": null, "currentItemEndArrowhead": "arrow", "scrollX": 583.2388916015625, "scrollY": 573.6323852539062, "zoom": { "value": 1 }, "currentItemRoundness": "round", "gridSize": null, "gridColor": { "Bold": "#C9C9C9FF", "Regular": "#EDEDEDFF" }, "currentStrokeOptions": null, "previousGridSize": null, "frameRendering": { "enabled": true, "clip": true, "name": true, "outline": true } }, "files": {} } ``` %%