[ECR]: Allow for alternate mediaTypes

[jdolitsky]

Tell us about your request

Hello. I'd like to request that ECR allow for arbitrary mediatypes specified in image manifests (config mediaType and layer mediaType).

Alternatively, AWS can keep a public whitelist of mediaTypes allowed in ECR and engage the community to request support for new types.

Which service(s) is this request for?

ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

The OCI distribution spec is currently being stretched to support more than just container images (Helm charts, OPA bundles, etc). In order for us to reason about these things, the mediaType field of manifests is being replaced with new custom values.

When running an out-of-the-box instance of docker distribution (2.7+), these new mediaTypes are allowed to be pushed.

However, when pushing these things to ECR, the user is met with a 404 Not Found. Please see oras-project/oras#105 where this issue was first reported.

To give more context, a manifest pushed using docker CLI looks like this:

{
    "schemaVersion": 2,
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json", // <-----
        "size": 7023,
        "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
    },
    "layers": [
...

Whereas a manifest pushed by Helm 3 looks like this:

{
    "schemaVersion": 2,
    "config": {
        "mediaType": "application/vnd.cncf.helm.chart.config.v1+json", // <-----
        "size": 2,
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
    },
    "layers": [
...

It appears that ECR is whitelisting mediaTypes allowed in the registry.

Are you currently working around this issue?

No. The only current solution (while staying on AWS) is to run a custom instance of docker/distribution on an EC2 or in a container.

Additional context

Interest is growing in this concept and we would love to be able to point users to use the registry tied to their cloud provider (AWS) vs. running a custom registry.

Additionally, Azure Container Registry (ACR) allows for this, and maintains a public list of mediaTypes with first-class support.

Attachments

Here are a few links that have some more info on the topic:

• "Cloud Native Artifact Registries evolve from Docker Container Registries" blog post
• ORAS: CLI/Go lib for pushing/pulling alternate artifacts
• bundle.bar: A free service that hosts new types of lightweight, cloud-native artifacts

---

I appreciate your time, thank you!

[jdolitsky]

There is also another report in Helm slack seeing a 403 instead:

[...] status="403 Forbidden" url="https://xxx.dkr.ecr.us-east-2.amazonaws.com/v2/testpush/blobs/sha256:22568708069eb08fd477d4785eeb2c60ac1f09c40e1f9cd674250fbf1c8284f6"
Error: unexpected response: 403 Forbidden
helm.go:56: [debug] unexpected response: 403 Forbidden

[jtoberon]

Thanks for the feedback. We're aware of the OCI distribution spec proposal, and we attend the weekly OCI calls to track its progress. We will continue to participate to help drive the distribution spec in a direction that we believe is useful to our customers, and if it addresses their needs we will be happy to adopt it over time.

[eonicle]

+1. Would really like to see both charts and container images in one place.

[dmytroleonenko]

Almost a year now. Is it so hard to implement?

[apenney]

I'm definitely disappointed I can't publish helm charts into ECR at this point. We're going to have to build and maintain additional infrastructure as a result. I hope that AWS can prioritize this (as it doesn't look like a ton of development and is massively beneficial to people that use helm).

[omieomye]

We're starting to look at this with a view to support, and would love to hear if there use cases for more than Helm charts that should publish to ECR, and what problem that would help solve.

[danbeaulieu]

@omieomye This is a good starting point to see the direction OCI is going: https://github.com/containers/image/pulls?q=is%3Apr+mediaType+

CNAB mediaType: application/vnd.docker.distribution.manifest.v2+json, application/vnd.cnab.config.v1+json

Another good source is containerd: https://github.com/containerd/containerd/blob/master/images/mediatypes.go

The criu/checkpoint/restore ones would be extremely useful.

I've run in to a few efforts of using wasm as extensions (envoy proxy filters), with those bundles being stored in oci registries. Example "application/vnd.wasm.content.layer.v1+wasm" from https://radu-matei.com/blog/wasm-to-oci/

[RichiCoder1]

Shipping Helm & CNAB bundles through the registry, plus future support for WASM packages also are motivating factors for us wanting this too.

[gerred]

Shipping kudo.dev bundles as artifacts is also on our roadmap.

[crsantini]

This addition will be very appreciated. We have clients with several compliance cases where ECR ticks all the boxes and wouldn't require us to look to validate other tools like ChartMuseum

[vchan2002]

We're also looking into using Porter.sh and it would be really great if ECR supports CNAB!

https://porter.sh/compatible-registries/

[omieomye]

We've been working on this, and looking to release OCI artifact support soon.

[mhausenblas]

Alright, following up here on what Bob mentioned in his Container Day keynote earlier today. From now on, you can give this feature a try in ap-southeast-1.

For now, focus on Helm charts using something like the following:

# preparation:
CHART_TAG=helm-ehw-0
REPO_NAME=artifact-test
REPO_URI=$(aws ecr create-repository --repository-name $REPO_NAME | jq -r .repository.repositoryUri)
REPO_CHART=$REPO_URI:$CHART_TAG
REGISTRY=$(echo $REPO_CHART | sed s%\/$REPO_NAME:$CHART_TAG%%)

# enable feature:
export HELM_EXPERIMENTAL_OCI=1

# prep Helm chart locally:
helm create example-hello-world
helm chart save example-hello-world $REPO_CHART

# push to ECR:
aws ecr get-login-password | helm registry login -u AWS --password-stdin $REGISTRY
helm chart push $REPO_CHART

# verify:
$ aws ecr batch-get-image \
           --repository-name $REPO_NAME \
           --image-ids imageTag=$CHART_TAG \
           --query 'images[].imageManifest' --output text
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:d10882dfb2c97a6a8ff886f42a174a475822b3aa1a6bc48c67e4ddc6183ef2a6",
    "size": 153
  },
  "layers": [
    {
      "mediaType": "application/tar+gzip",
      "digest": "sha256:5f0823edaeed5d40a1dffeb5029454426ab2e4626c39c999ceba9661f035d6ce",
      "size": 3591
    }
  ]
}

We will follow up with more details here and the complete set of supported OCI artifacts in Amazon ECR within a week (give or take), so keep an eye on this issue here to learn more.

[bit-cloner]

Would be great to see artifacts like npm, maven and binaries are also supported.
GCP's artifact registry just started supporting npm and maven
https://cloud.google.com/artifact-registry/docs/supported-formats

[fmedery]

Would be great to see artifacts like npm, maven and binaries are also supported.
GCP's artifact registry just started supporting npm and maven
https://cloud.google.com/artifact-registry/docs/supported-formats

Hello,
AWS CodeArtifact works with commonly used package managers and build tools like Maven and Gradle (Java), npm and yarn (JavaScript), or pip and twine (Python).

[mhausenblas]

We shipped the feature now:

• docs
• launch blog post

[omieomye]

Closing.

[javirln]

Hello, sorry to comment on a closed issue.

I've just wanted to know if custom OCI config media types is something supported at the moment on AWS ECR? We are aiming to push custom OCI artifacts to AWS ECR that we can do right now but, only if the default mediaType for the config file is used: application/vnd.unknown.config.v1+json.

It basically means that it's failing every time we try to push an artifact with a custom mediaType.

[TBBle]

What is the custom config media type you're trying to push? Does it match the spec application/[registration-tree].[org|company|entity].[objectType].[optional-subType].config.[version]+[optional-configFormat]?

[javirln]

Hello @TBBle !

Yes it does, it's something like: application/vnd.vmware.product.subproduct.config.v1+json. The content of the file config is {} and the name of the file config. Getting the error:

failed commit on ref \"manifest-sha256:c5efb084aec06ab6e276cbbb229aaec87ea088742d120305dca49c7f9c7f730a\": unexpected status: 405 Method Not Allowed

[TBBle]

That appears to be complaining about the manifest, not the config? Or at least, assuming your code is derived from or using https://github.com/containerd/containerd/blob/v1.5.5/remotes/handlers.go#L55-L86 or similar, it thinks it's a manifest locally, or has been told that explicitly by WithMediaTypeKeyPrefix. I'm assuming the connection based on hitting this error message which is in the push code path.

My guess is that the failure came from the manifest push, when AWS saw the config media type it didn't like for some reason, rather than the config push. One way to distinguish that would be use use application/vnd.vmware.product.subproduct.config.v1 as your type and not push a config, that would probably fail the same way. The blog post demonstrates a config JSON that's {} so what you're doing now looks prima-facie fine.

405 Method Not Allowed is a weird failure to get though, as it's only allowed for DELETE, unless we are actually using the wrong method for a given endpoint for some reason, which would be a weird thing to vary by config.mediaType.

An OCI registry may include more details in the body of a a 4xx error, but sadly the code implementing stringification of this error doesn't show that.

Anyway, at this point I'm wild-ass guessing, and a concrete repro case, e.g., if you can fake up something that has the same problem using containerd and ctr content push-object, or a repro with oras, would go a long way to helping track down if this is a user error, ECR error, or issue in the implementation of pushing in containerd (or whatever you're using, if it's not containerd, or oras which uses containerd libraries for this code-path too).

[javirln]

Wow thanks @TBBle for the detailed explanation!

We are indeed using oras (0.11.1) to push the custom artifacts which is pretty simple, one config file and one layer with random text, also pretty straightforward. I've managed to narrow down the error to the following, taking into account the following snippet:

memoryStore := content.NewMemoryStore()
layer := memoryStore.Add("simple-component", "application/vnd.vmware.product.subproduct.layer.v1.octet-stream", []byte("random-text"))
pushContents := []ocispec.Descriptor{layer}

configContent := []byte("{}")
config := ocispec.Descriptor{
	MediaType:   "application/vnd.vmware.product.subproduct.config.v1+json",
	Digest:      digest.FromBytes(configContent),
	Size:        int64(len(configContent)),
}

if _, err := oras.Push(context.Background(), getDockerResolver(), "ref:latest", memoryStore, pushContents, oras.WithConfig(config)); err != nil {
 // ...

That will work and, which would be the same as adding directly oras.WithConfigMediaType("the-type"), as well. If using the memoryStore to create the descriptor, it will populate an additional field Annotations with:

org.opencontainers.image.title: "Name of the file"

Adding any annotation to the config seems to fail. Haven't check it with recent versions of oras though.

[TBBle]

Yeah, that's a bug in ORAS. Per the Annotation rules:

• Keys using the org.opencontainers.image namespace are reserved for use in the OCI Image Specification and MUST NOT be used by other specifications and extensions, including other OCI specifications.

So AWS is correctly rejecting that descriptor because it has an annotation which is only allowed to be used with these media types.

That said, 405 is a confusing error to return in this case, and by the spec, it probably should be 400 with a body explaining that the annotation is not allowed.

---

Oops, reading back, you mentioned that any annotation failed. That's an AWS implementation bug, since Annotation is definitely allowed in a Descriptor.

I'm not sure if anyone from AWS is actually looking at this, but now that we can see a definite bug, perhaps it can be raised in a new issue here, or as a support ticket somewhere.