[ECR]: Pull-Through Cache Helm Support

[kamirendawkins]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Support Helm Charts with ECR Pull-Through Cache.

Which service(s) is this request for?
ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
With the Karpenter v0.17.0 release the chart has permanently moved to the ECR Public Gallery. While our current deployment mechanisms support Helm Charts stored in Private ECR Repositories it does not support Public ECR Repositories. ECR Pull-Through Cache is a natural fit for this but it would appear to not be supported.

Are you currently working around this issue?
We are actively working on a solution that is low maintenance, ie does not require us to constantly pull and update private ECR Repositories.

Additional context
I cannot seem to find any examples of this which is what leads me to believe this is not currently supported, if that is not true than this issue should be closed out.

Attachments
N/A

[jlbutler]

Hi @kamirendawkins thanks for the issue! Can you describe your current workflow for supporting Helm charts stored in private ECR repositories? I believe that PTC should work as you have indicated, and we might need to update some docs to make that clear. But first, I'd like to confirm we don't have a gap in our understanding. Thanks!

[kamirendawkins]

Hi @jlbutler , sure thing.

Currently our CI System refreshes ECR tokens periodically to ensure that Helm can pull the chart. This is and has been working for a while. The issue that we are seeing with PTC is that it will work on the first pull, the private repository is created but unlike a container image pull it does not appear cache anything. The only reason we are using PTC is due to our CI system not supporting public OCI repositories at the moment, not an ECR specific problem.

We see the same symptoms locally and in our CI system where on the first pull it will grab the tarball of the chart but on every subsequent pull we get the following:

helm pull oci://REDACTED_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/ecr-public/karpenter/karpenter --version v0.17.0 --destination /tmp/testing
Error: manifest does not contain minimum number of descriptors (2), descriptors found: 1

To test we have tried this with both of the following AWS owned public charts:

• https://gallery.ecr.aws/karpenter/karpenter
• https://gallery.ecr.aws/aws-controllers-k8s/s3-controller

[jlbutler]

Thanks again for opening this issue and for the additional info. We are looking into this and also seeing unexpected behavior. We will continue to look into this and post back here when we have more info!

[jlbutler]

Quick update on this. We have identified the issue here and are actively working on addressing it. We'll post back here again when we have an idea of when a fix will be available - and we'll be adding some additional tests. All hopefully will be soon. Thanks again @kamirendawkins for taking the time to open this issue, much appreciated.

[kamirendawkins]

@jlbutler Any update on this issue by any chance?

[jlbutler]

@kamirendawkins sorry for the delay in responding! This is currently rolling out and should be available in all regions in a matter of days. I'll check back in here once the rollout is complete, thanks again!

[jlbutler]

Deployment has completed, marking this shipped.

$ helm pull oci://<acctid>.dkr.ecr.us-west-2.amazonaws.com/ecr-public/karpenter/karpenter --version v0.20.0 --destination /tmp/testing

Pulled: 510431938379.dkr.ecr.us-west-2.amazonaws.com/ecr-public/karpenter/karpenter:v0.20.0
Digest: sha256:13ac5e23172091e50463e1b977ec200cca29bcb388cf11c7483ff11d8b3a081e

Thanks again @kamirendawkins !

[erfanw]

Hi @jlbutler,
What kind of configurations are needed for ECR to be able to pull-through public helm charts? Does it support every upstream?
Thanks

[jstewart612]

@erfanw

https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html

Amazon ECR currently supports creating pull through cache rules for the following upstream registries.

Docker Hub, Microsoft Azure Container Registry, GitHub Container Registry, and GitLab Container Registry (Requires authentication)

Amazon ECR Public, the Kubernetes container image registry, and Quay (Doesn't require authentication)

The implication of this statement is that ECR pullthrough cache can only pull from these remotes. So, no, it does not support "every upstream". As for the public/private designation, you can configure authentication or not on the above, and that's going to govern whether your ECR pullthrough cache instance can pull public or private images. Remember: OCI is an image format. As such, your given Helm Chart would essentially have to meet two conditions to be usable here:

• Has to be hosted in the above supported registries
• Helm chart has to be distributed using the OCI Image and Distribution specification (Chartmuseum which a lot of public charts still use won't work here)

I hope that helps. I find that, for what ECR doesn't support, I'll just internally mirror into my Github and publish OCI to GHCR and then pull that. Not ideal to have to mirror everything, but with the fairly wide support of ECR pullthrough cache registry support, the frequency I do so reduces more and more with time.