Default image output from buildx v0.10 cannot run on Google Cloud Run or AWS Lambda

[jedevc]

As of Buildx 0.10, images are built with provenance -- this behavior is opt-out, and can be disabled with --provenance=false if using the buildx build cli, or provenance: false if using docker/build-push-action.

Images built with these default attestations that use the attestation storage from buildkit cannot be run on Google Cloud Run or AWS Lambda. While both Cloud Run and Lambda support the OCI format, they do not support multi-platform images.

From Google Cloud Run docs (emphasis mine):

Cloud Run accepts container images in the Docker Image Manifest V2, Schema 1, Schema 2, and OCI image formats.

Manifest lists used for Multi-Architecture Images are not supported.

Note

Cloud Run has now added support for processing image indexes: #1533 (comment).

From Lambda docs (emphasis mine):

Lambda provides multi-architecture base images. However, the image you build for your function must target only one of the architectures. Lambda does not support functions that use multi-architecture container images.

Ideally, GCR and Lambda should support multi-platform images, and detect the current platform from the Docker manifest list / OCI index.

As a temporary workaround (in order of preference):

• Users should set --provenance=false on buildx build, or set provenance: false on docker/build-push-action
• Users can force an explicit buildx version to v0.9.1
• Users can force an explicit buildkit version to v0.10

[steren]

Cloud Run PM here, we received many reports about this. At the moment, Cloud Run doesn't support multi architecture images.

We were tracking this work item, but we do not have an delivery date to communicate yet.

[piotrekkr]

@steren If you are Cloud Run PM then I have a request that can help save few hours of debugging for everyone. Would be nice to return proper error message when we try to use multi-platform images with cloud run instead of returning

Image 'europe-docker.pkg.dev/xxx/xxxx' not found

It clearly exist in artifact registry since I choose it from select list when creating cloud run service.

Message like:

Image 'europe-docker.pkg.dev/xxx/xxxx'  is a multi-platform image and is not supported

Would probably reduce debug time to 15 min instead of 6h.

Thanks

[davideme]

@steren and buildx team, Cloud Run supports SLSA Build level 3 through provenance, see https://cloud.google.com/software-supply-chain-security/docs/sds/deploy-run-view-security-insights#build
Is there a way to support provenance in buildx without multi architecture images?
Or the way provenance is added by Google Cloud Build and buildx is not the same?

[billinghamj]

Cloud Run was supporting the old Docker "fat manifest" manifest lists application/vnd.docker.distribution.manifest.list.v2+json - we've been using them for over a year

It just doesn't support the new OCI format

[billinghamj]

I would also note that the Docker documentation seemingly isn't up to date on covering this change: https://docs.docker.com/registry/spec/manifest-v2-2/

[majidakbaridh]

Is there any way to use newer version of setup-buildx-action but changing mediaType to "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json" ?

[tonistiigi]

@majidakbaridh you can use -o oci-mediatypes=false but if you want to avoid attestations then disable attestations directly with --provenance=false. You can't have the attestations without the OCI mediatypes as well.