Unable to use helm-secrets in ArgoCD with age encryption

[LvffY]

Current Behavior

Hi

First of all, thanks for this repo and all its documentation ! 🎉

For now, I'm stuck with some weird error. I'm trying to setup the use of helm-secrets within my ArgoCD implementation.

I'm using init container method and mount secrets method. Based on this, I'm trying to use helm secrets with an age encryption.

For this, I have the following files :

• ArgoCD helm charts values:

repoServer:
  env:
    ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
    - name: HELM_PLUGINS
      value: /custom-tools/helm-plugins/
    - name: HELM_SECRETS_SOPS_PATH
      value: /custom-tools/sops
    - name: HELM_SECRETS_KUBECTL_PATH
      value: /custom-tools/kubectl
  volumeMounts:
    ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
    - mountPath: /custom-tools
      name: custom-tools
    ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume
    - mountPath: /helm-secrets-private-keys/
      name: helm-secrets-private-keys

  # -- Additional volumes to the repo server pod
  volumes:
    ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
    - name: custom-tools
      emptyDir: {}
    ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume
    - name: helm-secrets-private-keys
      secret:
        secretName: helm-secrets-private-keys
  initContainers:
    ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
    - name: download-tools
      image: alpine:latest
      command: [sh, -ec]
      env:
        - name: HELM_SECRETS_VERSION
          value: "3.9.0"
        - name: SOPS_VERSION
          value: "3.7.1"
        - name: KUBECTL_VERSION
          value: "1.22.0"
      args:
        - |
          mkdir -p /custom-tools/helm-plugins
          wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

          wget -qO /custom-tools/sops https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux
          wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl

          chmod +x /custom-tools/*
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

• Private helm charts with encrypted values

• ArgoCD application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: spark-dem-soa-1
  namespace: argocd
  # Finalizer that ensures that project is not deleted until it is not referenced by any application
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: myRepo
    targetRevision: feature/add-argocd
    path: spark/spark
    helm:
      version: v3
      releaseName: spark-app
      valueFiles:
        - secrets+age-import:///helm-secrets-private-keys/key.txt?values_enc.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: dem-soa-1
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Because I don't have any sensitive values from (just make some tests), the setup using the unencryted values works

• Unencrypted value

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: spark-dem-soa-1
  namespace: argocd
  # Finalizer that ensures that project is not deleted until it is not referenced by any application
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: myRepo
    targetRevision: feature/add-argocd
    path: spark/spark
    helm:
      version: v3
      releaseName: spark-app
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: dem-soa-1
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

When I'm installing my app with encryted values, I get the error open secrets+age-import:///helm-secrets-private-keys/key.txt?values_enc.yaml: no such file or directory.

I made several tests and :

• The command run by argocd works locally but not on the server when made manually (so this seems not to be a "server error")
• The secrets key file is well mounted on reposerver
• The repo is well pulled
• SOPS commands are returning the correct values
• helm secrets template commands (when the environment variable SOPS_AGE_KEY_FILE is set) are returning the correct values

I'll try

Expected Behavior

When installing my app, ArgoCD should be able to decrypt my value file correctly using age.

Steps To Reproduce

No response

Environment

• Keycloak Version: [e.g. 10.0.1]
• keycloak-config-cli Version: [e.g. 1.4.0]
• Java Version: [e.g. 11]

Anything else?

I'll try to use gpg to see if I have a different results

[jkroepke]

Hey @LvffY ,

I recently add the age support 2 days ago. While the documentation says 3.9.0 minimum, the age support is included in 3.10.0.

You can update your helm-secrets version to 3.10.0 and retest you scenario?

[LvffY]

Hello @jkroepke

Thanks for the (super-)fast answer :)

Just upgrading the helm-secrets version has resolved my problem.

May be an upgrade for the documentation could avoid you more issues :)

[jkroepke]

I updated the docs. Thanks for the feedback!

[RashRAJ]

hello am also facing this issue but I am using KMS encryption,

• ArgoCD helm charts values:

reposerver:
  replicas: 1
  serviceAccount:
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam::****************:role/argocd_repo_server_iam_role

  env:
    - name: HELM_PLUGINS
      value: /custom-tools/helm-plugins/
    - name: HELM_SECRETS_SOPS_PATH
      value: /custom-tools/sops
    - name: HELM_SECRETS_VALS_PATH
      value: /custom-tools/vals
    - name: HELM_SECRETS_KUBECTL_PATH
      value: /custom-tools/kubectl
    - name: HELM_SECRETS_CURL_PATH
      value: /custom-tools/curl
    - name: HELM_SECRETS_VALUES_ALLOW_SYMLINKS
      value: "true"
    - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
      value: "true"
    - name: HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL
      value: "true"
  volumes:
    - name: custom-tools
      emptyDir: {}
  volumeMounts:
    - mountPath: /custom-tools
      name: custom-tools

  initContainers:
    - name: download-tools
      image: alpine:latest
      command: [sh, -ec]
      env:
        - name: HELM_SECRETS_VERSION
          value: "4.1.1"
        - name: KUBECTL_VERSION
          value: "1.24.6"
        - name: VALS_VERSION
          value: "0.18.0"
        - name: SOPS_VERSION
          value: "3.7.3"
      args:
        - |
          mkdir -p /custom-tools/helm-plugins
          wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

          wget -qO /custom-tools/sops https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux
          wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl

          wget -qO- https://github.com/variantdev/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals;
          
          # helm secrets wrapper mode installation (optional)
          # RUN printf '#!/usr/bin/env sh\nexec %s secrets "$@"' "${HELM_SECRETS_HELM_PATH}" >"/usr/local/sbin/helm" && chmod +x "/custom-tools/helm"
          
          chmod +x /custom-tools/*
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

• Argocd Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-hello-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: test
    server: https://kubernetes.default.svc
  project: default
  sources:
  - repoURL: https://github.com/Helium-Health/staging-gitops.git
    targetRevision: main
    ref: values
  - repoURL: https://github.com/Helium-Health/Helium-Helm-Chart.git
    targetRevision: main
    path: ./
    helm:
      valueFiles:
        #- $values/staging/test-api/values.yaml
        #- secrets: $values/staging/test-api/values.secrets.yaml
         - secrets://staging/test-api/values.secrets.yaml 
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

[RashRAJ]

failed exit status 1: Error: open secrets://staging/test-api/values.secrets.yaml: no such file or directory

[jkroepke]

ArgoCD decide to implement multi-source apps which are not compatible with helm-secrets.

argoproj/argo-cd#11866

[RashRAJ]

Is there a workaround you could suggest?

[jkroepke]

Avoid the multi-source feature

[RashRAJ]

Hi @jkroepke this is my new manifest, but I am still getting failed exit status 1: Error: open secrets://secrets.yaml: no such file or directory

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-hello-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: test
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://github.com/Helium-Health/Helium-Helm-Chart.git
    targetRevision: main
    path: ./
    helm:
      valueFiles:
        #- $values/staging/test-api/secrets.yaml
        #- secrets: $values/staging/test-api/secrets.yaml
        - secrets://secrets.yaml 
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true