OCI images pushed to dockerhub don't show up and aren't available

[johnmcollier]

Problem description

When I build and push OCI images using the buildah CLI to dockerhub, they get pushed fine (and can be pulled with buildah pull ...), but they won't show up in the Dockerhub UI, nor can they be pulled by the docker CLI or Kubernetes (using Docker). But Docker does appear to see that the image exists

docker pull:

johns-mbp-3:~ johncollier$ docker pull johncollier/spring-push-test
Using default tag: latest
Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

buildah pull

sh-4.2# buildah pull johncollier/spring-push-test
Getting image source signatures
Copying blob edf72af6d627 skipped: already exists
Copying blob 8d3eac894db4 skipped: already exists
Copying blob 3e4f86211d23 skipped: already exists
Copying blob d72258b97fd0 skipped: already exists
Copying blob efaa572fa153 skipped: already exists
Copying blob f7277927d38a skipped: already exists
Copying blob 4b9886dd7157 skipped: already exists
Copying blob f81d814c0c2c skipped: already exists
Copying blob 3dcc376f5183 skipped: already exists
Copying config 7765f9ec00 done
Writing manifest to image destination
Storing signatures
7765f9ec00b0adc6cc1469535241b886deaf0b69e16617837d7608627bf6f3da

This seems to be a recent change, as up until yesterday, using the exact same version of buildah, everything worked fine and OCI images could be pushed to DockerHub using buildan and they could be pulled using Kubernetes/Docker (and seen in the DockerHub UI).

I should also note that other Docker registries work perfectly fine without issue (such as Artifactory, OKD's internal registry, etc). I can push images to them using buildah and Kubernetes can pull them perfectly fine.

[manishtomar]

@johnmcollier Thanks for the report. Yesterday hub registry was upgraded to latest version that supports OCI images. In your case it looks like until yesterday buildah was pushing docker images and NOT OCI images probably because it detected that OCI support was not there. Yesterday after the upgrade it pushed an OCI image which docker does not support pulling as of now. Hence the following error:

Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

However we are aware of hub UI not showing tags behind OCI images. We are working on getting this fixed and should release it (hopefully) sometime next week. LMK if you still have questions.

[kedmison]

I am getting the same docker pull issue with the recent docker-ce and docker-registry.

# docker pull localhost:5000/kelvin/centos-bashecho
Using default tag: latest
Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

# docker version
Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:28:55 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:34 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[manishtomar]

@kedmison As I mentioned earlier docker-ce does not support pulling OCI images as of now. It only supports pulling docker images.

[johanbrandhorst]

This affects any images built by podman since it uses buildah under the hood.

[thaJeztah]

Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

That problem looks the same as moby/moby#39727, which is fixed by moby/moby#40070, and will be back ported to docker 19.03 for the next (19.03.5) patch release (backport is here: docker-archive/engine#405)

[CameronNemo]

Just spent a lot of time trying to deal with this... would really be nice if the docker client could handle OCI images, or even if there were a workaround available.

[hSATAC]

@CameronNemo I feel you bro

[wipatrick]

just upgraded to Docker 19.03.5 on my Jetson Nano as pointed out by @thaJeztah and it seems to work.

[kallisti5]

Still happening as of today. Pushing a latest image with podman results in repo "not showing up" via the ui.

Putting the repo into the url bar actually shows a weird broken repo @ hub.docker.io.

[fedelibre]

@kallisti5 I had the same problem, but after some hours everything showed up in docker hub. I also did a pull from another machine in the meanwhile.. don't know if it may be related.

[manishtomar]

Hub UI does not show OCI images. We are aware of the issue and are working towards fixing this. Hoping to do this in a few weeks.

[CameronNemo]

Having diverse image building tools is quite valuable. Thank you for the work you have done so far. Looking forward to the UI being updated. In the meantime, I am twiddling my thumbs while I wait for the gitlab.com shared runners to update their docker version.

[lexfrei]

@manishtomar any news?

[TriplEight]

Tried pushing the image built with buildah without --format=docker. It appears in the docker hub interface

but looks weird, do not show the links, OS/ARCH and LAST PULL fields.

Also can confirm that both buildah pull and docker pull work with this image.

My guess is there's something to do with the manifest format.

[github-actions]

We are clearing up our old issues and your ticket has been open for 6 months with no activity. Remove stale label or comment or this will be closed in 15 days.

[belidzs]

This is still an issue as of today. A workaround with podman would be to push with the --format=docker option

[github-actions]

We are clearing up our old issues and your ticket has been open for 6 months with no activity. Remove stale label or comment or this will be closed in 15 days.

[tnk4on]

This issue has been fixed by docker/roadmap#262.