Helm through Kustomize with a private Helm repository - x509 certificate signed by unknown authority

[gaeljw]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When using Helm charts through Kustomize, with a private Helm repository and private CA, ArgoCD fails to sync application.

To Reproduce

Create an ArgoCD ApplicationSet that targets a git repository, I guess that it's not specific to ApplicationSet, a simple Application would do the same but in my case it's an ApplicationSet.

The resulting Application looks like:

project: ...
source:
  repoURL: 'http://gitlab.mycompany.net/xxx.git'
  path: path/on/git/repo
  targetRevision: HEAD
destination:
  namespace: ...
  name: ...

This git repository contains a Kustomize folder with following kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: private-chart
    repo: https://artifactory.mycompany.net/artifactory/helm
    version: 0.4.0

When synchronizing, the following error occurs:

rpc error: code = Unknown desc = `kustomize build .path/on/git/repo --enable-helm` failed exit status 1: Error: Error: looks like "https://artifactory.mycompany.net/artifactory/helm" is not a valid chart repository or cannot be reached: Get "https://artifactory.mycompany.net/artifactory/helm/index.yaml": x509: certificate signed by unknown authority : unable to run: 'helm pull --untar --untardir .path/on/git/repo/charts --repo https://artifactory.mycompany.net/artifactory/helm private-chart --version 0.4.0' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-342675645/helm HELM_CACHE_HOME=/tmp/kustomize-helm-342675645/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-342675645/helm/.data] (is 'helm' installed?)

Expected behavior

The App should sync without error in ArgoCD.

Other informations

1/ ArgoCD is setup with our certs configured in argocd-tls-certs-cm as per https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#repositories-using-self-signed-tls-certificates-or-are-signed-by-custom-ca

2/ Note that using the same Helm repository directly without going through Kustomize does work, for instance:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: external-secrets-operator
  namespace: argocd
spec:
  generators: ...
  template:
    metadata:
      name: '{{cluster}}-external-secrets-operator'
    spec:
      project: ...
      source:
        chart: external-secrets
        repoURL: https://artifactory.mycompany.net/artifactory/helm
        targetRevision: 0.7.2
      destination:
        name: '{{cluster}}'
        namespace: external-secrets

Also note that I did not add the Helm repo as an ArgoCD repo, I believe it's not necessary. But if that helps, I could do it.

Version

{
    "Version": "v2.6.7+5bcd846",
    "BuildDate": "2023-03-23T14:57:27Z",
    "GitCommit": "5bcd846fa16e4b19d8f477de7da50ec0aef320e5",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.5.7 2022-08-02T16:35:54Z",
    "HelmVersion": "v3.10.3+g835b733",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.19.1"
}

Links

This looks somehow related to the following issues but not entirely sure it's the same problem:

• ApplicationSet controller receives x509: certificate signed by unknown authority, when using SCM, PR or Git generator with private bitbucket server #9607
• Helm repositories with self signed certs doesn't work when used as umbrella charts #3693
• helm chart dependencies fail to download from private repo #6477

Edit: this is a more similar duplicate of:

• Controller doesn't trust certificates set in ArgoCD when fetching external resources #12256
• Application using Kustomize with Helm cannot access private remote helm repo #10745

[gaeljw]

I tried the workaround mentionned at #9607 (comment) but in the argocd-application-controller which is the one raising the error in my case but without luck. Adding the volume + volumeMount did not make the trick.

[gaeljw]

I confirm the workaround defined at #6477 (comment) works.

In my case, I reused the argocd-tls-certs-cm with a Kustomize patch to mount it in argocd-server, repo-server and application-controller:

- op: "add"
  path: "/spec/template/spec/containers/0/volumeMounts/0"
  value:
    mountPath: /etc/ssl/certs
    name: tls-certs-workaround
- op: add
  path: /spec/template/spec/volumes/0
  value: 
    name: tls-certs-workaround
    configMap:
      name: argocd-tls-certs-cm
      items:
        - key: artifactory.mycompany.net
          path: ca-certificates.crt

patches:
- path: certificate-patch.yaml
  target:
    kind: StatefulSet
    name: argocd-application-controller
- path: certificate-patch.yaml
  target:
    kind: Deployment
    name: argocd-server
- path: certificate-patch.yaml
  target:
    kind: Deployment
    name: argocd-repo-server

[todaywasawesome]

Closing in favor of existing issue: #6477