ArgoCD fails to deploy Helm charts from a GitHub OCI registry using pre-release version wildcards

[gilles-gosuin]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

ArgoCD fails to deploy Helm charts from a GitHub OCI registry using pre-release version wildcards.

To Reproduce

• Publish a helm chart to GitHub OCI registry using a version such as 1.0.0-beta.2
• Make sure that it works when specifying the exact version (it does)
• Try and deploy it using a version wildcard. I have tried using >=1.0.0-0, ^1.0.0-0 and ^1.x.x-0; weirdly enough, all fail with an authentication error:

rpc error: code = InvalidArgument desc = application spec for ***** is invalid:
InvalidSpecError: Unable to generate manifests in .:
rpc error: code = Unknown desc = unable to get tags:
failed to get tags: GET "https://ghcr.io/v2/***/***/tags/list":
GET "[https://ghcr.io/token?scope=repository%253A*****%252F*****%253Apull&service=ghcr.io](https://ghcr.io/token?scope=repository%253A*****%252F*****%253Apull&service=ghcr.io)":
response status code 401: unauthorized: authentication required

Expected behavior

I'm expecting that it works as well as when I do a helm pull (which works fine):

$ helm registry login ghcr.io/*****
Username: anonymous
Password:
Login Succeeded

$ helm pull oci://ghcr.io/*****/***** --version "1.0.0-beta.2"
Pulled: ghcr.io/*****/*****:1.0.0-beta.2
Digest: sha256:3f30205f451e4e5327dff145e8912158e3295057ec092a74e8fb3be540c8019c

$ helm pull oci://ghcr.io/*****/***** --version "^1.0.0-0"
Pulled: ghcr.io/*****/*****:1.0.0-beta.2
Digest: sha256:3f30205f451e4e5327dff145e8912158e3295057ec092a74e8fb3be540c8019c

$ helm pull oci://ghcr.io/*****/***** --version "^1.x.x-0"
Pulled: ghcr.io/*****/*****:1.0.0-beta.2
Digest: sha256:3f30205f451e4e5327dff145e8912158e3295057ec092a74e8fb3be540c8019c

$ helm pull oci://ghcr.io/*****/***** --version ">=1.0.0-0"
Pulled: ghcr.io/*****/*****:1.0.0-beta.2
Digest: sha256:3f30205f451e4e5327dff145e8912158e3295057ec092a74e8fb3be540c8019c

Version

argocd: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.9.1+58b04e5

[gris-gris]

I am encountering a similar issue related to the usage of wildcard versions in Helm charts. Specifically, when setting the 0.*.* version of the Helm chart, the credentials for the OCI Helm repository are being disregarded, resulting in a 401 response:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = unable to get tags: failed to get tags: GET "https://harbor.example.com/v2/helm-charts/demo/tags/list": response status code 401: unauthorized: unauthorized to access repository: helm-charts/demo, action: pull: unauthorized to access repository: helm-charts/demo, action: pull

Upon switching the repository to Public, the pull operation works seamlessly, and the application retrieves the latest OCI Helm image.

Furthermore, when specifying an exact version such as 1.0.7, the process works as intended

[aarozhkov]

+1 With GAR and pre-release tags feature.

Does not matter how/what type of authentication was added for private repo
If your repo looks like https://host:port/**path_to_exact_repo** - pre-release feature will not work properly.

I can point to root cause in Oras-go library: it does not support repos with path part

Example

argocd repo add --project myargoproject--name my-helm-oci --type helm --enable-oci us-west1-docker.pkg.dev/myproject/my-helm-repo

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  project: myargoproject
  source:
    chart: myapp
    repoURL: us-west1-docker.pkg.dev/myproject/my-helm-repo/
    # targetRevision: ">=0.0.0-0"
    helm:
      passCredentials: true
      parameters:
        - name: somevalue
          value: awesomeValue
  destination:
    name: mycluster
    namespace: mynamespace

Some findings

Code flow for helm OCI repos with pre-release:

argo-cd/util/helm/client.go

Line 403 in b12630c

func (c *nativeHelmChart) GetTags(chart string, noCache bool) (*TagsList, error) {

It uses Oras-go library and do such initialization with c.repoURL == us-west1-docker.pkg.dev/myproject/my-helm-repo for for credentials func StaticCredential

argo-cd/util/helm/client.go

Lines 431 to 438 in b12630c

	repo.Client = &auth.Client{
	Client: client,
	Cache: nil,
	Credential: auth.StaticCredential(c.repoURL, auth.Credential{
	Username: c.creds.Username,
	Password: c.creds.Password,
	}),
	}

Tag list fetched with Oras library in 2 stages:

• try to do tags requests without any auth headers
• Get an answer from server with auth instructions( challenge Www-Authenticate ) and try to authenticate with these instructions:
  https://github.com/oras-project/oras-go/blob/main/registry/remote/auth/client.go#L221-L262

Problem here: Oras try to prepare an Authorization token for host part of repository URL, but credentials are saved against full URL and get EmptyCredentials instead of provided on initial state:

https://github.com/oras-project/oras-go/blob/main/registry/remote/auth/client.go#L180
https://github.com/oras-project/oras-go/blob/main/registry/remote/auth/client.go#L258
https://github.com/oras-project/oras-go/blob/main/registry/remote/auth/client.go#L293-L294
Where credentials is a function created on initialization stage with

https://github.com/oras-project/oras-go/blob/main/registry/remote/auth/client.go#L65-L78

if hostport == registry {}
"us-west1-docker.pkg.dev" != "us-west1-docker.pkg.dev/myproject/my-helm-repo"

As a possible solution from Argo-CD side: Use host part during oras initialization instead of full repoURL.
Due this initialization scoped only by thisfunction it should not affect other requests/repos located on same host.

[thepabloaguilar]

Hey @gris-gris and @aarozhkov! The authentication issue should be fixed now, could you test it please?

I've tested it locally and it seems to be working as expected now!

[gilles-gosuin]

I can confirm that resolving pre-release version ranges in 2.10.4 now works as expected.

Thanks! ☺️

[NITHIN-JOHN-GEORGE]

Hi @gilles-gosuin I am getting same issue in argocd v2.11.3 for GAR