Helm in Argo CD does not use added self-signed CA for pulling Helm Charts

[dan-m8t]

Checklist:

• [ x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ x] I've included steps to reproduce the bug.
• [ x] I've pasted the output of argocd version.

Describe the bug

I am trying to use Harbor as oci compliant Helm repository, the certificate of that registry is selfsigned.
In the docs it is shown how to add and trust custom CAs. (either via cli or UI, I tried both)
When I try to roll out a helm chart with a dependency pointing to Harbor Argo CD throws an Helm error:

Error: failed to do request: Head "https://myharbor.customtld/v2/something/chart/whatever/manifests/1.0.11": x509: certificate signed by unknown authority

I tried to pull myself the above Chart locally, I trusted the root CA of my Harbor and that just works fine

To Reproduce

• add self-signed TLS certificate or CA (tried both seperate and as a chain)
• add Helm Chart repository with self-signed certificate
• try to roll it out or:
• kubectl exec -t argocd-repo-server-xyz -- helm pull oci://myharbor.customtld/something/chart/whatever --version 1.0.11
• Helm inside Argo CD throws x509 certificate authority unknown

Expected behavior

• Argo CD should trust the certificate for helm pull actions

Version

argocd: v2.6.0+unknown
  BuildDate: 2023-02-06T22:47:06Z
  GitCommit:
  GitTreeState:
  GitTag: 2.6.0
  GoVersion: go1.20
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.4.7+81630e6
  BuildDate: 2022-07-18T21:17:35Z
  GitCommit: 81630e6d5075ac53ac60457b51343c2a09a666f4
  GitTreeState: clean
  GoVersion: go1.18.4
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
  Helm Version: v3.8.1+g5cb9af4
  Kubectl Version: v0.23.1
  Jsonnet Version: v0.18.0

[fengshunli]

Whether to skip certificate verification @crenshaw-dev

[dan-m8t]

Helm does support --insecure-skip-tls-verify for pull actions but I need the argocd-repo-server to know it somehow.
Maybe via ENV ?

Edit:

Even this is not working inside the repo pod:

helm pull --insecure-skip-tls-verify oci://myharbor.customtld/something/chart/whatever --version 1.0.11
Error: failed to do request: Head "https://myharbor.customtld/v2/something/chart/whatever/manifests/1.0.11": x509: certificate signed by unknown authority

This worked fine on my local machine, but my Helm version on my machine is quite recent.

Helm in the argocd-repo-server pod:

helm version version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}

my local version:

version.BuildInfo{Version:"v3.11.0", GitCommit:"472c5736ab01133de504a826bd9ee12cbe4e7904", GitTreeState:"clean", GoVersion:"go1.19.5"}

Maybe I should update Argo CD, but it is still an issue on how to set --insecure-skip-tls-verify for helm pull actions or even better trust my certificate.

[dan-m8t]

Anything I can do or help with?

[dan-m8t]

bump

[Diliz]

Having the same issue here, did you find a way to fix it @dan-m8t ?

[dan-m8t]

Unfortunately not yet, I tried the workaround mentioned in similar issues here with mounting the hosts CA folder to Argo CD. No luck yet. I'm a little bit irritated that no developer answers :(

[joeltraber]

I face the same issue, I also added the certificate to argocd as trusted, it is also mapped into the container correctly, can be found somewhere in app/config/tls/..data/

no workaround found so far
(using Kustomize to inflate the helm chart)

[koshrf]

I tried everything and it is just an argocd thing that needs to be fixed at code level but I don't even know where to start.

I added the TLS certificates to the pods, you can add the OCI harbor repo to argocd and it does login, but when it goes to do the pull it just fails with the certificate error, it isn't using the certificates and not using the insecure tls flag.

[simonoscr]

bump

[mboutet]

#8508 seems to indicate it should have been fixed, but apparently not :/

[mboutet]

I'm not well versed in Go and I don't know the codebase at all, but it seems that part of the fix would be to put the logic that injects the --insecure-skip-tls-verify, --ca-file, etc. flags from:

argo-cd/util/helm/cmd.go

Lines 188 to 228 in 42bdb5a

	func (c *Cmd) Fetch(repo, chartName, version, destination string, creds Creds, passCredentials bool) (string, error) {
	args := []string{c.pullCommand, "--destination", destination}
	if version != "" {
	args = append(args, "--version", version)
	}
	if creds.Username != "" {
	args = append(args, "--username", creds.Username)
	}
	if creds.Password != "" {
	args = append(args, "--password", creds.Password)
	}
	if creds.InsecureSkipVerify && c.insecureSkipVerifySupported {
	args = append(args, "--insecure-skip-tls-verify")
	}
	args = append(args, "--repo", repo, chartName)
	if creds.CAPath != "" {
	args = append(args, "--ca-file", creds.CAPath)
	}
	if len(creds.CertData) > 0 {
	filePath, closer, err := writeToTmp(creds.CertData)
	if err != nil {
	return "", err
	}
	defer argoio.Close(closer)
	args = append(args, "--cert-file", filePath)
	}
	if len(creds.KeyData) > 0 {
	filePath, closer, err := writeToTmp(creds.KeyData)
	if err != nil {
	return "", err
	}
	defer argoio.Close(closer)
	args = append(args, "--key-file", filePath)
	}
	if passCredentials && c.helmPassCredentialsSupported {
	args = append(args, "--pass-credentials")
	}
	return c.run(args...)

in:

argo-cd/util/helm/cmd.go

Lines 231 to 240 in 42bdb5a

	func (c *Cmd) PullOCI(repo string, chart string, version string, destination string) (string, error) {
	return c.run(
	"pull",
	fmt.Sprintf("oci://%s/%s", repo, chart),
	"--version",
	version,
	"--destination",
	destination,
	)
	}

[pasha-codefresh]

I was able to reproduce this issue, looks like it requires two things

1. Upgrade helm to 3.11.3+
2. Provide --insecure-skip-tls-verify inside PullOCI function

[halvorstein]

Facing the same issue.
Any updates on that matter?

[randallweytens]

This is still an issue even with v2.10.7. Been beating my head on this for a month. Updating the argocd-tls-certs-cm does nothing to fix this. Having to inject certificates into the container trust store doesn't seem like a sustainable fix.

[flypenguin]

i'd say this needs to be reopened – we're experiencing the same issue on v2.9.3, the fixes are apparently part of v2.6.x.

@crenshaw-dev – you closed this, maybe have a 2nd look? :)

[SimonWoidig]

I have encountered this issue as well. I am using Argo 2.10 with declarative setup.
I have added the CA cert to ArgoCD certificates. I also added the OCI registry as a repository. It shows a green mark next to it, meaning it was connected successfully.
I cannot create an app manually using this OCI repo - it just shows a TLS error. If I declare the application via the declarative method, it just shows it errored out.
See the images for more.

[randallweytens]

I was able to fix this by mounting a config map that contained our certificate chain. I can supply the specifics hopefully tomorrow.On Aug 7, 2024, at 09:15, Šimon Woidig ***@***.***> wrote: I have encountered this issue as well. I am using Argo 2.10 with declarative setup. I have added the CA cert to ArgoCD certificates. I also added the OCI registry as a repository. It shows a green mark next to it, meaning it was connected successfully. I cannot create an app manually using this OCI repo - it just shows a TLS error. If I declare the application via the declarative method, it just shows it errored out. See the images for more. image.png (view on web) image.png (view on web) image.png (view on web) image.png (view on web) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[svghadi]

Seems like related to #19138. It mainly happens if helm repo is created using path in the URL.