Unable to Pull Helm Charts from ECR Repositories

[romitzz1]

This is in response to /pull/362. That have helm charts in the root of ECR repos, and flux is unable to pull them.

The error code we now get: invalid chart reference: failed to get chart version for remote reference: could not get tags for "SERVICENAME": could not fetch tags for "oci://ACCOUNTNUMBER.dkr.ecr.REGION.amazonaws.com/SERVICENAME": GET "https://ACCOUNTNUMBER.dkr.ecr.REGION.amazonaws.com/v2/SERVICENAME/tags/list":` credential required for basic auth`

source-controller version: ghcr.io/fluxcd/source-controller:v0.32.1

This is the URL of our HelmRepository : url:oci://ACCOUNTNUMBER.dkr.ecr.REGION.amazonaws.com``

Complete error: {"level":"error","ts":"2022-12-09T20:46:20.623Z","msg":"invalid chart reference: failed to get chart version for remote reference: could not get tags for \"SERVICENAME\": could not fetch tags for \"oci://ACCOUNTNUMBER.dkr.ecr.REGION.amazonaws.com/SERVICENAME\": GET \"https://ACCOUNTNUMBER.dkr.ecr.REGION.amazonaws.com/v2/SERVICENAME/tags/list\": credential required for basic auth","name":"default-SERVICENAME","namespace":"flux-system","reconciler kind":"HelmChart","annotations":null,"error":"InvalidChartReference","stacktrace":"github.com/fluxcd/pkg/runtime/events.(*Recorder).AnnotatedEventf\n\tgithub.com/fluxcd/pkg/runtime@v0.24.0/events/recorder.go:137\ngithub.com/fluxcd/pkg/runtime/events.(*Recorder).Eventf\n\tgithub.com/fluxcd/pkg/runtime@v0.24.0/events/recorder.go:114\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.RecordContextualError\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:56\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:193\ngithub.com/fluxcd/source-controller/controllers.(*HelmChartReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/helmchart_controller.go:228\ngithub.com/fluxcd/source-controller/controllers.(*HelmChartReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/helmchart_controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}

We have been running flux successfully using a custom source controller built from this (1a4b57a) commit version.

[darkowlzz]

Hi @romitzz1, thanks for providing more details from the comment in #362 . I'm a little confused by the description and your patch that's working for you. Do you no longer have the issue after using your patch? The patch essentially achieves the same result of allowing parsing of registry address, instead of expecting the address to have an image name and tag at the end. The same test case that you have passing in your patch is also present in #362 . #362 makes the address validation more relaxed. So it should just work without your patch.
If HelmChart OCI failed due to the login code from #362, the error message would have "failed to get credential from..." based on https://github.com/fluxcd/source-controller/blob/v0.32.1/controllers/helmchart_controller.go#L525. But the error you shared seems to be coming from source-controller helm chart remote builder https://github.com/fluxcd/source-controller/blob/v0.32.1/internal/helm/chart/builder_remote.go#L131 .

Can you please clarify if you still have the issue or it's working fine with that patch you have?

Also, please see fluxcd/source-controller#951 for charts at the root of ECR. It's still a known issue and the workaround for now is to not have it at the root.

[darkowlzz]

Upon further testing, I think I found the root cause of the issue you have and that also answers why your patch must have worked. The fix in #362 changed how the registry value was being derived for AWS. But it was already a bug for other providers. Your patch didn't include that change which made it work for you.
Please see #434 for more details about the issue. That'll fix it for all the cloud providers when providing repository root address.
Thanks again for reporting and bringing attention to this.

[darkowlzz]

Fixed in #434

Please leave a comment if it's still an issue.