Configure account_key as secret for Loki with Azure storage

[lucasoares]

To use Loki with Azure I need to set an account_key attribute when using Helm.

storage_config:
  azure:
    account_name: ???
    account_key: ???
    container_name: loki
    request_timeout: 0

Shouldn't we be able to to set this setting from an environment variable or a secret? Currently I need to keep my accountKey inside my values.yaml file and this is not secure.

Grafana Tempo, for example, accepts the AZURE_STORAGE_KEY variable. I tried to use it with Loki and had the following error:

prefix:index/
restype:container
timeout:31'.
   Code: AuthenticationFailed
   GET REDACTED?comp=list&delimiter=&prefix=index%2F&restype=container&timeout=31
   Authorization: REDACTED
   User-Agent: [Azure-Storage/0.14 (go1.20.1; linux)]
   X-Ms-Client-Request-Id: [12239412-105b-43bd-66cd-1799e064bc05]
   X-Ms-Date: [Thu, 13 Apr 2023 20:13:59 GMT]
   X-Ms-Version: [2020-04-08]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Content-Length: [767]
   Content-Type: [application/xml]
   Date: [Thu, 13 Apr 2023 20:13:58 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [d4d87f47-601e-0021-3044-6e7fd1000000]

Is there any workaround?

[slyt]

The same issue exists for s3 storage configuration as well. secretAccessKey and accessKeyId should be able to reference Kubernetes Secrets instead of being provided in plaintext in values.yaml.

[slyt]

I got this working for s3 storage by setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in a secret then use extraEnvFrom: to inject them into environment variables for each pod. This works because Loki uses the AWS SDK client which prioritizes environment variables for credentials. Example config below inspired by this comment.

The same approach might work for Azure (I'm unable to test with Azure). Based on the Azure Go SDK docs, you can set AZURE_CLIENT_ID and AZURE_CLIENT_SECRET via environment variables.

helm values:

write: 
  extraEnvFrom:
    - secretRef:
        name: loki-s3-credentials # contains AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
read:
  extraEnvFrom:
    - secretRef:
        name: loki-s3-credentials
backend:
  extraEnvFrom:
    - secretRef:
        name: loki-s3-credentials

Secret config:

apiVersion: v1
kind: Secret
metadata:
  name: loki-s3-credentials
type: Opaque
data:
  AWS_ACCESS_KEY_ID: YOUR_ACCESS_KEY_ID_HERE
  AWS_SECRET_ACCESS_KEY: YOUR_AWS_SECRET_ACCESS_KEY_HERE

Alternatively, you could try using environment variable expansion in the configuration by passing in -config.expand-env=true via extraArgs and use the ${VAR} syntax in the config. I was unable to get this working with s3, it wasn't actually expanding the variables in the config in the pods (checked with kubectl exec -it loki-write-0 -- cat /etc/loki/config/config.yaml). I am unable to test with Azure. Example config below in case someone else wants to give it a try.

helm values (untested):

loki:
  storage:
    bucketNames:
      chunks: chunks
      ruler: ruler
      admin: admin
    type: azure
    azure:
      accountName: ${AZURE_ACCOUNT_NAME}
      accountKey: ${AZURE_ACCOUNT_KEY}
      useManagedIdentity: false
      useFederatedToken: false
      userAssignedId: null
      requestTimeout: null
read:
  extraArgs:
    - "-config.expand-env=true"
write:
  extraArgs:
    - "-config.expand-env=true"
backend:
  extraArgs:
    - "-config.expand-env=true"

Secret config:

apiVersion: v1
kind: Secret
metadata:
  name: loki-azure-credentials
type: Opaque
stringData:
  AZURE_ACCOUNT_NAME: YOUR_AZURE_ACCOUNT_NAME_HERE
  AZURE_ACCOUNT_KEY: YOUR_AZURE_ACCOUNT_KEY_HERE

[faustodavid]

Thanks @slyt, I have tested it, and it works! 🎉

[sinacek]

apiVersion: v1
kind: Secret
metadata:
  name: loki-azure-credentials
type: Opaque
data:
  AZURE_ACCOUNT_NAME: YOUR_AZURE_ACCOUNT_NAME_HERE
  AZURE_ACCOUNT_KEY: YOUR_AZURE_ACCOUNT_KEY_HERE

@slyt please change example to use stringData: instead of data:. I'm use to use helm and missed that there is missing base64. Thanks for all

[lausserl]

Hi, I tried to configure the secrets for an azure blob storage the last few days and ran into issues with the configuration as described.
I have tried this configuration

helm values (untested):

loki:
  storage:
    bucketNames:
      chunks: chunks
      ruler: ruler
      admin: admin
    type: azure
    azure:
      accountName: ${AZURE_ACCOUNT_NAME}
      accountKey: ${AZURE_ACCOUNT_KEY}
      useManagedIdentity: false
      useFederatedToken: false
      userAssignedId: null
      requestTimeout: null
read:
  extraArgs:
    - "-config.expand-env=true"
write:
  extraArgs:
    - "-config.expand-env=true"
backend:
  extraArgs:
    - "-config.expand-env=true"

Secret config:

apiVersion: v1
kind: Secret
metadata:
  name: loki-azure-credentials
type: Opaque
stringData:
  AZURE_ACCOUNT_NAME: YOUR_AZURE_ACCOUNT_NAME_HERE
  AZURE_ACCOUNT_KEY: YOUR_AZURE_ACCOUNT_KEY_HERE

I tried with config.expand-env=true and with the secretRef, but I always get illegal base64 data at input byte 0.

I have tried it with putting the AZURE_ACCOUNT_KEY directly into the helm chart, this works well, but as soon as I use a secret, I get a base64 error.

Do you have any suggesetions on why this could still be?

[Aangbaeck]

@lausserl Hi, did you solve it? I have the exact same issue. I have put many hours into trying to solve this but could not.

[Sargastico]

@slyt @lausserl @Aangbaeck I am facing the same problem using the loki-stack chart. Do any of you have any idea how to configure the values.yaml file to save the logs on Azure Storage? I can't keep any secrets on my values.yaml.

Like @Aangbaeck, I've also have put many hours into trying to solve this but could not.

[faustodavid]

Hi, for me the explanation from @slyt works with Azure Storage.

This is how I did it using AzureKeyVaultSecret.

Step 1: Store Secrets in Azure Key Vault

Ensure your Azure storage account name and key are securely stored in an Azure Key Vault.

Step 2: Define AzureKeyVaultSecret in Helm Template

Go to the Helm Templates directory and create a file that defines the AzureKeyVaultSecret. This configuration creates Kubernetes secrets from the Azure Key Vault and outputs two environment variables, AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_KEY.

Here's the Helm template for the Azure Key Vault secret:

# Template for the storage account key
apiVersion: spv.no/v1
kind: AzureKeyVaultSecret
metadata:
  name: loki-az-storage-account-key
  namespace: {{ .Release.Namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "-5"
spec:
  vault:
    name: {{ .Values.kvAzName }}
    object:
      name: {{ .Values.kvAzStorageAccountKey }}
      type: secret
  output:
    secret:
      name: loki-az-storage-account-key
      dataKey: AZURE_ACCOUNT_KEY

---

# Template for the storage account name
apiVersion: spv.no/v1
kind: AzureKeyVaultSecret
metadata:
  name: loki-az-storage-account-name
  namespace: {{ .Release.Namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "-5"
spec:
  vault:
    name: {{ .Values.kvAzName }}
    object:
      name: {{ .Values.kvAzStorageAccountName }}
      type: secret
  output:
    secret:
      name: loki-az-storage-account-name
      dataKey: AZURE_ACCOUNT_NAME

Step 3: Configure values.yaml for Each Environment

For each environment, such as staging or production, update the values-{env}.yaml file with the following parameters:

kvAzName: "YOUR_KEYVAULT_NAME"
kvAzStorageAccountKey: "THE_SECRET_NAME_FOR_ACCOUNT_KEY"
kvAzStorageAccountName: "THE_SECRET_NAME_FOR_ACCOUNT_NAME"

Step 4: Update Loki's values.yaml

Update the values.yaml file used by Loki's Helm chart with the secrets that will be injected:

loki:
  storage:
    azure:
      accountName: ${AZURE_ACCOUNT_NAME}
      accountKey: ${AZURE_ACCOUNT_KEY}

And in each Loki component add:

extraArgs:
   - "-config.expand-env=true"
extraEnvFrom:
  - secretRef:
      name: loki-az-storage-account-name
  - secretRef:
      name: loki-az-storage-account-key

I hope this help! 🙏🏼

[Sargastico]

For anyone looking for a solution using loki-stack chart, the problem is that apparently declaring the argument this way:

extraArgs:
   - "-config.expand-env=true"

Doesn't work and ends up resulting in a manifest with corrupted flags, like this:

spec:
  affinity: {}
  containers:
    - args:
        - '-config.file=/etc/loki/loki.yaml'
        - '-0=-config.expand-env=true'

Somehow, -0= is added to the arg, which is causing the container to crash.

What solved it for me was defining the yaml values like this:

  extraArgs:
    "-config.expand-env": "true"
  extraEnvFrom:
    - secretRef:
        name: loki-storage-secrets
    - secretRef:
        name: loki-storage-secrets

• extraArgs value bugged in tempo 1.6.1? helm-charts#2705

[Sargastico]

still unable to get this working due to the illegal base64 data at input byte 0 error reported by @lausserl

[Sargastico]

Problem solved! Now I'm using the basic loki helm chart. My setup:

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: monitoring

helmCharts:
  - name: loki
    repo: "https://grafana.github.io/helm-charts"
    version: "5.36.2"
    releaseName: loki
    namespace: monitoring
    valuesFile: values.yaml

values.yaml

loki:
  auth_enabled: false
  schema_config:
    configs:
    - from: "2020-12-11"
      index:
        period: 24h
        prefix: index_
      object_store: azure
      schema: v11
      store: boltdb-shipper
  storage:
    bucketNames:
      chunks: loki
    type: azure
    azure:
      accountName: ${AZURE_ACCOUNT_NAME}
      accountKey: ${AZURE_ACCOUNT_KEY}
  extraArgs:
    - "-config.expand-env=true"
  extraEnvFrom:
    - secretRef:
        name: loki-storage-secrets
write:
  extraArgs:
    - "-config.expand-env=true"
  extraEnvFrom:
    - secretRef:
        name: loki-storage-secrets
read:
  extraArgs:
    - "-config.expand-env=true"
  extraEnvFrom:
    - secretRef:
        name: loki-storage-secrets
backend:
  extraArgs:
    - "-config.expand-env=true"
  extraEnvFrom:
    - secretRef:
        name: loki-storage-secrets
test:
  enabled: false
monitoring:
  dashboards:
    enabled: false
  selfMonitoring:
    enabled: true
    grafanaAgent:
      installOperator: false
  lokiCanary:
    enabled: false

[npuichigo]

The document of Loki should be enhanced like Tempo (https://grafana.com/docs/tempo/latest/configuration/hosted-storage/azure/#sample-configuration-for-tempo-distributed-mode) and mention the need to specify extraArgs and extraEnv configuration for all the services.