[ECR] [request]: public registry tag listing

[rpnguyen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Enable discovering tags of a public repository, possibly by implementing the /tags/list endpoint https://github.com/opencontainers/distribution-spec/blob/master/spec.md#content-discovery

Which service(s) is this request for?
ECR Public

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Discover the available tags for a repository I don't own

Are you currently working around this issue?
Finding the list via the ECR public gallery

[zhonghui12]

To be clear, my feature request is to let a public user who doesn't have the permisson to log into the repo use the CLI like aws ecr-public describe-image-tags so he/she can get all the tags of a public repo.

[Infra-Red]

This just got me, my CI system is failing while listing image tags:

❯ curl -i -H "Authorization: Bearer $TOKEN" https://public.ecr.aws/v2/xxxxxxx/awesome-image/tags/list
HTTP/2 404
date: Fri, 19 Feb 2021 15:26:51 GMT
content-type: text/plain; charset=utf-8
content-length: 19
docker-distribution-api-version: registry/2.0
x-content-type-options: nosniff

404 page not found

I assumed that ECR Public implements Docker Registry HTTP API V2, but apparently it is not 🤷‍♂️

[deppy]

I would also point out that the documentation for public registries sounds like the Amazon ECR Public supports the entire Docker Registry HTTP API.

Amazon ECR Public supports the Docker Registry HTTP API.

[vgrigoruk]

It is very disappointing to see that the official documentation says:

Amazon ECR Public supports the Docker Registry HTTP API. However, you must provide an authorization token with every HTTP request.

But at the same time, the list-image-tags API is not supported (even if I am calling the API using the TOKEN that belongs to ECR registry owner):

TOKEN=$(aws ecr-public get-authorization-token --region us-east-1 --output=text --query 'authorizationData.authorizationToken')
curl -i -H "Authorization: Bearer $TOKEN" https://public.ecr.aws/v2/registry_alias/repository_name/tags/list

[davidspek]

Just wanted to leave a note here as not being able to list the image tags for images on the Public Gallery is causing problems with dependency management for parts of the Kubeflow project.

[dekimsey]

We ran into this issue as well as we sync the repository to local registry for performance and stability reasons. I will be escalating the issue through our AWS representative and would recommend others do the same.

[simingweng]

It seems this isn't an issue for private ECR repository https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html

I'm able to invoke the "tag/list" http endpoint on one of my private repository but not public ones.

[mbamber]

We have been looking at a way to poll for updates to specific repositories so as to remain up-to-date (specifically we are hoping to receive notifications when a new version of the App Mesh Envoy container is pushed).

This would be really easy if there was a programmatic way to list tags for a repository 😢

[lorengordon]

This is impacting Dependabot support for ECR Public also, dependabot/dependabot-core#4212.

[pierluigilenoci]

@maishsk is there any chance to see this problem addressed?

[pierluigilenoci]

@srrengar @saleemmx any update on this topic?

[bambaryla]

Hello, having helm 3.8 released with OCI support, looks like AWS Public ECR is not compliant with dependencies https://helm.sh/docs/topics/registries/#specifying-dependencies. For me "helm depepndency update" fails exactly with the same error of not being able to list tags.

[arunsollet]

Thank you for raising this issue. Amazon ECR Public now supports listing of tags in a public repository.

[jalaziz]

Seems like the listing tags API is not OCI compliant? Seems like a big mistake on AWS's part to not adhere to the OCI spec.

The OCI spec mandates that the response MUST be in the following format:

{
  "name": "<name>",
  "tags": [
    "<tag1>",
    "<tag2>",
    "<tag3>"
  ]
}

The response from the tag list API does not follow this format and means that it cannot be used by OCI-compliant tooling.

The OCI spec also expects a Link response header instead of nextToken in the response body.

[dekimsey]

Oh wow, that's entirely and completely different API but sitting on the oci path, which makes it purposefully incompatible. I didn't think to double-check. Nice find. This is utterly useless for any reasonable use-case and misses the heart of this issue entirely.

I'll be sure to flag this with our account rep. I would recommend others do the same.

[lorengordon]

Yikes. That's a big miss. Please re-open and fix the response format for this API.

[arunsollet]

Hi everyone, this was released prematurely and does not yet reflect our intended feature set. The OCI compliant tags API is coming soon and we will share more details here once it is fully and finally released. Apologies for the confusion.

[dekimsey]

Thank you @arunsollet for clarifying, that's really good to hear!

[jlbutler]

Hi everyone 👋

We are seeking guidance around the functionality of this API and how strictly we need to adhere to the OCI spec in order to meet immediate needs. We intend for full OCI compliance, but if we consider iterating toward that compliance we can likely solve most blocking issues more quickly.

The main issue with the premature release is that it was not yet complete (as some observed the internal-only response body). It seems to me the main thrust of solving most use cases a) the api must exist, and b) the response body must be formatted as specified.

Beyond this, I'd like to understand how important ordering and pagination are. The OCI spec calls for transparent tokens (in the form of tag values), and that tags returned are lexically order. There are some implementations which adhere strictly to the spec in this regard, and others that do not.

What do folks think about a release sooner to address the main issue first, and we can iterate toward sorting and potentially using a transparent token in a subsequent release? This maps to how the v2 tags/list API works on ECR today.

I know it's not perfect, but will this meet most use cases? Flux, ArgoCD, Skopeo, Dependabot are the main ones I'm thinking about, none of which depend upon pagination or ordering as far as I can tell.

To leave this as a simple question for folks to 👍 or 👎 - would you support if we defer lexical sorting and transparent tokens for pagination, if we can ship the API and properly formatted response body sooner?

Thanks very much!

[jlbutler]

Hey all!

We have started rolling this feature out and are already seeing a good deal of traffic on it. You may still see 404's on the endpoint until the rollout is complete, but wanted to let you all know since you may have some automation in place that starts working for the first time as this comes online.

As always - please let us know if anyone sees any issues or has questions. Otherwise, we'll post again once the deployments are complete.

Thanks again for all the patience and guidance, especially from the various open source maintainers we discussed this with.

[jlbutler]

This is now fully rolled out, please let us know if there are any issues/questions. Thanks again everyone!

> TOKEN=$(curl -k https://public.ecr.aws/token/ | jq -r '.token')

> curl -k -H "Authorization: Bearer $TOKEN"  https://public.ecr.aws/v2/amazonlinux/amazonlinux/tags/list | jq . | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3329    0  3329    0     0  16724      0 --:--:-- --:--:-- --:--:-- 17159
{
  "name": "amazonlinux/amazonlinux",
  "tags": [
    "2.0.20211201.0",
    "2.0.20220218.0-amd64",
    "2018.03.0.20201028.0",
    "2018.03.0.20220310.0",
    "2.0.20220316.0-arm64v8",
    "2.0.20220406.1",

[mattlorimor-bnch]

Can confirm. Dependabot on GitHub now updates public ECR FROM lines in Dockerfiles. Thanks!

[arununzer]

Hello @jlbutler ,, I am unable to do The Same for public.ecr.aws/docker/library/eclipse-temurin:17.0.6_10-jre-alpine

Can you help me with the Same ?

This is now fully rolled out, please let us know if there are any issues/questions. Thanks again everyone!

> TOKEN=$(curl -k https://public.ecr.aws/token/ | jq -r '.token')

> curl -k -H "Authorization: Bearer $TOKEN"  https://public.ecr.aws/v2/amazonlinux/amazonlinux/tags/list | jq . | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3329    0  3329    0     0  16724      0 --:--:-- --:--:-- --:--:-- 17159
{
  "name": "amazonlinux/amazonlinux",
  "tags": [
    "2.0.20211201.0",
    "2.0.20220218.0-amd64",
    "2018.03.0.20201028.0",
    "2018.03.0.20220310.0",
    "2.0.20220316.0-arm64v8",
    "2.0.20220406.1",