Helm AWS ECR OCI repository

[ronaldkan]

Hey there, just curious if anyone have tried using flux with ECR's private OCI helm repository?

I'm trying to leverage on a private ECR helm repository with IAM role and i'm constantly facing with
chart pull error: chart pull error: failed to get chart version for remote reference: GET "https://<accountID>.dkr.ecr.<region>.amazonaws.com/v2/helm-template/helm-template/tags/list": credential required for basic auth. Not sure if there's a way to turn off basic auth and use IAM role?

More Information:

HelmRepository object

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: ecr-helm-template
  namespace: flux-system
spec:
  type: oci
  interval: 5m0s
  url: oci://<accountId>.dkr.ecr.<region>.amazonaws.com/helm-template

HelmRelease object

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: test-app
  namespace: default
spec:
  chart:
    spec:
      chart: helm-template
      version: "0.0.1"
      sourceRef:
        kind: HelmRepository
        name: ecr-helm-template
        namespace: flux-system

[souleb]

The current implementation only supports credentials from secret as described in https://fluxcd.io/docs/guides/helmreleases/#helm-repository-authentication-with-credentials.

If you want to use IAM role, you would have to automate updating a secret with the generated token somehow I imagine.

[stefanprodan]

Like so https://fluxcd.io/docs/guides/cron-job-image-auth/#using-cronjob-to-sync-ecr-credentials-as-a-kubernetes-secret

[ronaldkan]

Awesome, thank you both for the prompt response!

[nalbury]

Hi sorry to tack onto an already answered discussion, but I'm currently trying to do this with a cronjob and while I can confirm the secret itself is being rotated and the new token is valid:

[root@jumbox ~]# kubectl get secret -n flux-system ecr-auth -o json |jq '.data.".dockerconfigjson"' -r |base64 --decode |jq '.auths."<redacted>.dkr.ecr.us-west-2.amazonaws.com/helm".password' -r |helm registry login -u AWS --password-stdin <redacted>.dkr.ecr.us-west-2.amazonaws.com/helm
Login Succeeded
[root@jumpbox ~]# helm show chart oci://<redacted>.dkr.ecr.us-west-2.amazonaws.com/helm/my-chart |grep apiVersion
apiVersion: v2

The source controller appears to use the old expired token until its restarted, at which point it reloads the new secret and can pull from ECR. Happy to open an issue somewhere but wasn't sure if this was expected given that the example provided I think creates a secret consumed by the image reflector vs the source controller here?
(edit: adding more output)

[root@jumpbox ~]# flux get source chart my-chart
NAME            REVISION	SUSPENDED	READY	MESSAGE
my-chart	0.2.0   	False    	False	chart pull error: chart pull error: failed to get chart version for remote reference: GET "https://<redacted>.dkr.ecr.us-west-2.amazonaws.com/v2/helm/my-chart/tags/list": unexpected status code 403: denied: Your authorization token has expired. Reauthenticate and try again.
[root@jumpbox ~]# kubectl delete pod -n flux-system source-controller-644c69fbf7-vpczd
pod "source-controller-644c69fbf7-vpczd" deleted
[root@jumpbox ~]# flux get source chart my-chart
NAME            REVISION	SUSPENDED	READY	MESSAGE
my-chart	0.2.0   	False    	True 	pulled 'my-chart' chart with version '0.2.0'

[stefanprodan]

Please open an issue in source-controller repo, it may be that we cache the secret.

[nalbury]

Will do, thanks!

[MatteoMori]

Hello @stefanprodan,

I am sorry but I was wondering if the above "cache" issue has been resolved?

Will it work as expected now?

[stefanprodan]

Yes please upgrade to Flux 0.32

[MartinEmrich]

Hi!

Should this work in Flux 0.35 now, without any workarounds (credentials/secrets)? I get the same error message (credential required for basic auth), yet the Kubernetes nodes (as always) have an appropriate AWS IAM Role to access the AWS ACR repository.

Possibly history repeats: fluxcd/image-reflector-controller#174 ff.

So I did some experiments:

As I understand, the source-controller is responsible for fetching OCI repository artifacts.
Also, as the error message looks identical to using the helm pull, I assumed it uses the same libraries under the hood. "helm" respects the docker CLI configuration files.

So I hacked together a modified image, including the AWS ECR credential helper (https://github.com/awslabs/amazon-ecr-credential-helper):

FROM ghcr.io/fluxcd/source-controller:v0.30.0
USER root
RUN mkdir /.docker
COPY config.json /.docker/config.json
RUN wget https://amazon-ecr-credential-helper-releases.s3.us-east-2.amazonaws.com/0.6.0/linux-amd64/docker-credential-ecr-login -O/usr/bin/docker-credential-ecr-login
RUN chmod +x /usr/bin/docker-credential-ecr-login
USER 65534

With that image, it works for me without juggling any credentials or secrets.

So the solution would be to include the docker credential helper for AWS ECR (and possibly equvalents for GCE, Azure etc.), providing seamless integration.

my .docker/config.json:

{
  "credsStore": "ecr-login"
}

would most certainly "surprise" users in heterogeneous environments, and wildcard support for registries is not (yet?) available: docker/cli#2928

Any chance to have at least the binary(ies) included in a future release? The docker client configuration file might be added at install time with a flux install --enable-ecr-whatever flag via a configMap?

[stefanprodan]

You need to set the provider to aws in the HelmRepository, see https://fluxcd.io/flux/components/source/helmrepositories/#aws

[MartinEmrich]

@stefanprodan thanks! That shall be my Facepalm Moment of today... :D

[stefanprodan]

Starting with v0.35.0, Flux has native support to authenticate automatically using the EKS worker node IAM role or IAM Role for Service Accounts (IRSA). Docs here: https://fluxcd.io/flux/components/source/helmrepositories/#aws

[fw-harish]

Please find the service account details...

harish@Harish-MacBook-Pro% kubectl get serviceaccount source-controller -o=jsonpath='{.metadata.annotations.eks.amazonaws.com/role-arn}'
arn:aws:iam::054331359164:role/cloud/cloud-workernodes-eu-central-1%

[stefanprodan]

Hmm why is there a % at the end?

[fw-harish]

I've no Idea but the actual role arn inside the SA looks good.

FYI: I've tried to replace the cloud-workernodes-eu-central-1 with worker_node_group1-eks-node-group-20230920075133937100000010. All the permissions attached to both roles were exactly same.

[stefanprodan]

Does the AssumeRoleWithWebIdentity have the source-controller service account specified? See the AWS docs https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html

[fw-harish]

Thanks a lot @stefanprodan. Actually that's the last missing piece of my puzzle.

[antoineDievDecath]

Hi

I followed steps to use IRSA to pull helm charts from ECR.
Source-controller log is:
failed to get credential from aws: could not validate OCI provider aws with URL oci://[my-ecr].dkr.ecr.eu-west-1.amazonaws.com

Flux installed with terraform 0.0.19
EKS 1.23
Source-controller v0.30.0
AWS_ROLE_ARN & AWS_STS_REGIONAL_ENDPOINTS ENV vars are presents

My HelmRepository:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: discovery
  namespace: kube-system
spec:
  interval: 5m0s
  url: oci://<my-ecr>.dkr.ecr.eu-west-1.amazonaws.com
  type: oci  
  provider: aws

What am I missing ?

[MartinEmrich]

maybe you're missing the repository name in the URL? Should be like url: oci://<my-account-id>.dkr.ecr.eu-west-1.amazonaws.com/<my-repository>

[antoineDievDecath]

That was my miss thx !

[antoineDievDecath]

There is something I don't get.
My helm-repository is ready when I set url: oci://<my-account-id>.dkr.ecr.eu-west-1.amazonaws.com/<my-repository>
But when I reference that repository in an HelmRelease with spec.chart.spec.chart: <my-repository> the source-controller try to fetch oci://<my-account-id>.dkr.ecr.eu-west-1.amazonaws.com/<my-repository>/<my-repository> and get a status code 404: name unknown.

My HelmRelease looks like that:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: discovery
  namespace: kube-system
spec:
  interval: 5m
  chart:
    spec:
      chart: discovery
      version: "22.06.197"
      sourceRef:
        kind: HelmRepository
        name: discovery
        namespace: kube-system
      interval: 1m
  values:
    #my_values

And as I understand the aws ecr documentation isn't possible to create "sub repository"

[MartinEmrich]

TBH, that confuses me, too, as I am a newbie to Helm in general. My observation so far:

AWS ECR Repository names are strings, possibly with slashes in between.

If I do helm push <chartname> oci://xxxx/<repositoryname>, I end up with <repositoryname>/<chartname> as the actual repository name.

so if you want to push an OCI helm chart named <chartname> to the AWS repository <repositoryname>, there is actually no <repositoryname>. Instead, the actual repositoryname is your <repositoryname>/<chartname>.
So you also have to aws ecr create-repository --repository-name <repositoryname>/<chartname> beforehand.

Then, to reference it again in a HelmRepository/HelmRelease, one has to "cut off" the chart name again from the full repository name and split them into HelmRelease and HelmRepository.

[antoineDievDecath]

Creating a repository named / and push into it my chart solved the problem. I think this is part of this helm improvement proposal https://github.com/helm/community/blob/main/hips/hip-0006.md#4-chart-names--oci-reference-basenames
Many Thanks to you for helping

[drzero42]

I must admit that this is also confusing me a lot. I can only get the HelmRepository to authenticate with the URL being oci://<my-account-id>.dkr.ecr.<my-region>.amazonaws.com/<reponame> but since <reponame> is where the chart is pushed to (meaning <reponame> == <chartname>), when I have a HelmRelease using that HelmRepo it will attempt to pull oci://<my-account-id>.dkr.ecr.<my-region>.amazonaws.com/<reponame>/<chartname>, which does not exist.

Can somebody clarify, with some easily understandable examples, what the correct way to achieve using IAM roles for ECR-based Helm repo is?

[stefanprodan]

@MartinEmrich my bad, that fix got into 0.25 but I forgot about it.

[cep21]

@MartinEmrich How did you get your helm chart to push with that URL? We can only use URLS like

oci://123456789.dkr.ecr.eu-central-1.amazonaws.com/chart-name

We were unable to make a ECR repository named helm-charts/chart-name and push there and when we asked AWS premium support, they said this format is not supported and the ECR repository name must exactly match the helm chart name.

[drzero42]

@cep21 Here is how we ended up doing it.

First we created a helm repo with aws ecr create-repository --repository-name helm/$APP_NAME.

During build we do helm push $APP_NAME-version.tgz oci://$AWS_ACCOUNT_ID.dkr.ecr.eu-central-1.amazonaws.com/helm.

Then in Flux we have this HelmRepository defined:

kind: HelmRepository
metadata:
  name: our-repo
  namespace: flux-system
spec:
  interval: 5m0s
  url: oci://AWS_ACCOUNT_ID.dkr.ecr.eu-central-1.amazonaws.com/helm
  type: "oci"
  provider: "aws"

That leaves us with being able to reference our chart:

kind: HelmRelease
metadata:
  name: api-app
spec:
  chart:
    spec:
      chart: APP_NAME
      version: "1.0.0"
      sourceRef:
        kind: HelmRepository
        name: our-repo
        namespace: flux-system

[MartinEmrich]

@cep21 The way I see it:

The AWS ECR (or other registry) repository name is "just a string", you can name it as you like (using the allowed characters, were / is one of them).

Helm instead made up this idea of a "chart name" within a "repository name". So a "ECR/OCI repository name" corresponds to "helm repository name" + "/" + "chart name". (I, too, find this counterintuitive, but I guess this is a legacy concept taken over from the standalone helm repositories?)

So if your helm repository name is "mycoolproject" and your chart name is "fancyfrontendcomponent", the resulting OCI/ECR/Harbor/whatever repository name is "mycoolproject/fancyfrontendcomponent".

thus e.g. helm push fancyfrontendcomponent oci://xxxx/mycoolproject requires the ECR repository mycoolproject/fancyfrontendcomponent

Hope that helps...

[cep21]

Thanks for the help. This really helped figure out why we couldn't push!