Configuring Loki to use AWS S3 Storage in DKP

Follow the instructions on this page to configure Loki/Velero to use AWS S3 Storage in DKP

Configuring Loki

The easiest way to get started with using AWS S3 storage with Grafana Loki is to use a set of static AWS credentials.

1. Execute the following command to get the namespace of your workspace:

   CODE

   dkp get workspaces

2. Set the WORKSPACE_NAMESPACE variable to the namespace copied in the previous step:

   CODE

   export WORKSPACE_NAMESPACE=<WORKSPACE_NAMESPACE>

3. Create a secret containing the static AWS S3 credentials. The secret is then mounted into each of the Grafana Loki pods as environment variables. 

   CODE

   kubectl create secret generic dkp-aws-s3-creds -n${WORKSPACE_NAMESPACE} \
   --from-literal=AWS_ACCESS_KEY_ID=<key id> \
   --from-literal=AWS_SECRET_ACCESS_KEY=<secret key>

4. Create a config overrides ConfigMap to update the storage configuration.
   NOTE: This can also be added to the installer configuration if you are configuring Grafana Loki on the Management Cluster.

   CODE

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: grafana-loki-overrides
     namespace: ${WORKSPACE_NAMESPACE}
   data:
     values.yaml: |
       loki:
         annotations:
           secret.reloader.stakater.com/reload: dkp-aws-s3-creds
         structuredConfig:
           storage_config:
             aws:
               s3: s3://<region>/<bucket name>
       ingester:
         extraEnvFrom:
           - secretRef:
               name: dkp-aws-s3-creds 
       querier:
         extraEnvFrom:
           - secretRef:
               name: dkp-aws-s3-creds 
       queryFrontend:
         extraEnvFrom:
           - secretRef:
               name: dkp-aws-s3-creds 
       compactor:
         extraEnvFrom:
           - secretRef:
               name: dkp-aws-s3-creds  
       ruler:
         extraEnvFrom:
           - secretRef:
               name: dkp-aws-s3-creds 
       distributor:
         extraEnvFrom:
           - secretRef:
               name: dkp-aws-s3-creds

5. Update the grafana-loki AppDeployment to apply the configuration override.
   NOTE: If you use the Kommander CLI installation configuration file, you don’t need this step

   CODE

   cat << EOF | kubectl -n ${WORKSPACE_NAMESPACE} patch appdeployment grafana-loki --type="merge" --patch-file=/dev/stdin
   spec:
     configOverrides:
       name: grafana-loki-overrides
   EOF

The reloader annotation only works in DKP 2.5.2+. Any changes to the AWS credential secret will not automatically reload some Loki pods. In this scenario, The ingester and compactor pods need to be manually restarted.

For more information, refer to https://github.com/grafana/helm-charts/issues/1905.