Problems with today's ubuntu containers on dockerhub

[DrDaveD]

I found this github repo by following the links on docker hub from the ubuntu images to doijanky to Jenkins to here.

When reading any of the ubuntu containers pushed to dockerhub today (it says 2 hours ago) using the go containerd library, the following error occurs:

unsupported image-specific operation on artifact with type "application/vnd.docker.container.image.v1+json

This error happens for example with podman and apptainer (formerly known as singularity), but it doesn't happen with docker. Can you help narrow down what changed to figure out what is causing the problem? Would it be possible to back out the current tags until the problem is resolved?

[tianon]

I'm confused, where are you getting application/vnd.docker.container.image.v1+json from? The published images are all OCI media types all the way down: https://explore.ggcr.dev/?image=ubuntu:22.04

The change you're seeing is that Ubuntu is now using OCI media types instead of Docker media types (#13950), and I'd be very surprised if podman and apptainer are not supporting OCI media types correctly (and I know for certain that containerd does, because it's what was used to push these in the first place 😅).

[DrDaveD]

Actually the error is coming from the github.com/containers/image/v5/docker library in the manifest package.

[DrDaveD]

That code was added in this PR which says in part of its description:

- Reject image-specific operations (like inspect, image ID) on OCI artifacts

[tianon]

Ahhhh, commented in #13950 (comment) with my discovery so the image maintainers hopefully see it:

The container is using the OCI media type for the manifest, but the objects inside (config and layer blob) are using Docker media types, and Docker is apparently more forgiving than other runtimes about the envelope and contents matching.

[tianon]

Anyone looking for content digests to pin to the older builds while this gets sorted out can find them in docker-library/repo-info@713e331

[paleozogt]

This does in fact break podman/buildah:

$ docker run \
   --rm -it --privileged quay.io/buildah/stable:v1.27.0 \
   buildah pull ubuntu:bionic
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:bionic...
copying system image from manifest list: parsing image configuration: unsupported image-specific operation on artifact with type "application/vnd.docker.container.image.v1+json"

[cjdcordeiro]

Yes, we're aware of it. It has indeed slipped through the cracks. We've fixed it and are now working on re-publishing new images. @tianon can either revert the #13950 or accept the new (whichever comes first)

[cjdcordeiro]

there's a new revision waiting for approval in #13988

the image being published here has already been published to other registries, and tools like podman seem to be happy now.

[tianon]

The fixed images are all pushed now. 👍

[paleozogt]

omg its working:

$ docker run \
     --rm -it --privileged quay.io/buildah/stable:latest \
     buildah pull ubuntu:bionic
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:bionic...
Getting image source signatures
Copying blob 72d9f18d70f3 done
Copying config 5d2df19066 done
Writing manifest to image destination
Storing signatures
5d2df19066aca89df8e5317544a1cb599dc657830184762ff6fdefaaf708db65