Pushing docker image to GPR: manifest invalid: Only schema version 2 is supported #25315

WyriHaximus · 2019-11-15T21:53:46Z

WyriHaximus
Nov 15, 2019

Since about an hour I’m getting the following error when I try to push a docker image tp GPR:

[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker.pkg.github.com registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
manifest invalid: Only schema version 2 is supported

Two hours ago everything work fine, the Dockerfile hasn’t changed, neither did the upstream image. Did Github change something an hour ago?

Answered by thepont

Nov 20, 2019

I might be stating things that people already know, But talking to github support they advised that the default storage for Github team is:

GitHub Team

Storage: 2GB
Data Transfer: 10GB

Also that the organisation storage can be checked by viewing: https://github.com/organizations/ORG/settings/billing

The following works for me, to list versions of packages on a project.

curl -X POST \
    -H "Accept: application/vnd.github.package-deletes-preview+json" \
    -H "Authorization: bearer TOKEN" \
    -d '{"query":"query {repository(owner:\"ORG\", name:\"REPOSITORY\") {registryPackages(last:20) {edges{node{id, name, versions(last:20){edges {node {id}}}}}}}}"}' \
    https://api.github.com/gr…

View full answer

jmahowald · 2019-11-16T21:04:54Z

jmahowald
Nov 16, 2019

I also had things working, then they started failing.

From here there did seem to be someone who found tagging with the git sha was problematic (too many characters).

https://github.saobby.my.eu.orgmunity/t5/GitHub-Actions/Can-t-push-to-GitHub-Docker-Registry-from-Actions/td-p/30665

0 replies

WyriHaximus · 2019-11-16T21:31:22Z

WyriHaximus
Nov 16, 2019
Author

Tried it with the following tags so I doubt it’s the tag langth: nieuwste, laatste, la

Also interesting detail is that I’m building the image I’m building this image on publicly on Github Actions and push it to Docker Hub. (Frequently, in this case the base image was less then a day old.)

0 replies

thepont · 2019-11-19T04:43:19Z

thepont
Nov 19, 2019

I have the same issue pushing to the Github repository but via CircleCI, can you update the version of docker client your using it the Github action for pushing… ( This may be a case of just updating the version of the GitHub action you’re using for pubishing)

I think its just because the old API has been deprecated and we are still pushing using older versions of the api ?

I updated my docker image used to publish my image to Docker version 19.03.5, build 633a0ea838 with no success, build is still failing with the metadata error.

0 replies

zymartinez · 2019-11-19T05:39:59Z

zymartinez
Nov 19, 2019

I tried your suggestion by trying to push outside of Github Actions. I’m currently on Docker version 19.03.2, build 6a30dfc on mac and I’m still getting the manifest invalid error. This may be caused by a different issue?

0 replies

thepont · 2019-11-19T05:41:47Z

thepont
Nov 19, 2019

Sorry, I just got mine building with another version and had the same issue again… Are people using the github private repo ?

I also don’t know if it’s worth noting that the docker manual says :

> When pushing images, clients which support the new manifest format should first construct a manifest in

> the new format. If uploading this manifest fails, presumably because the registry only supports the old

> format, the client may fall back to uploading a manifest in the old format.

This makes me think that it may be failing to upload using v2 then reverting to v1 where it then gets this error back from the server.

0 replies

WyriHaximus · 2019-11-19T06:28:09Z

WyriHaximus
Nov 19, 2019
Author

Yes I’m trying to push from a private repo to it’s private repo. This fallback to v1 could be it, there is a noticeable delay after the last layer is pushed before the error shows up.

Nevermind that, for both success and failure the delay is 6 seconds. Success gives you the latest digest SHA And failure that error.

0 replies

thepont · 2019-11-19T21:43:37Z

thepont
Nov 19, 2019

FYI, I sent a message detailing the issue github developer support, who have escalated the issue to the product team with the link to this community post.

I’ll update if I hear back from them.

Update, I figured out my issue:

So nested in the logs ( journalctl ) is the actual issue:

Nov 20 09:42:08 blue-pont dockerd[427]: time="2019-11-20T09:42:08.376878027+11:00" level=warning msg="failed to upload schema2 manifest: unauthorized: This operation would exceed the storage allotment for this account. - falling back to schema1"
Nov 20 09:42:08 blue-pont dockerd[427]: time="2019-11-20T09:42:08.376989057+11:00" level=warning msg="[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker.pkg.github.com registry NOW to avoid future disruption."
Nov 20 09:42:10 blue-pont dockerd[427]: time="2019-11-20T09:42:10.260794961+11:00" level=error msg="Not continuing with push after error: manifest invalid: Only schema version 2 is supported"

Seems our account is over the storage allocation ( I’m not the certain what this limit is ) and therefore we are no longer able to upload our images.

It is attempting to upload using schema 2 then failing and falling back, docker does not know how to handle the error from the github private repo and is therefor falling back in a reall confusing manner.

So i am not certain if we can delete older packages to free up space or what the limit is.

I used journalctl | grep dockerd (since I aways forget the flags) to get the full logs on Arch Linux, this will be different on OSX (https://docs.docker.com/docker-for-mac/troubleshoot/) or windows (https://docs.docker.com/v17.12/docker-for-windows/troubleshoot/).

But Docker will log the reason it falled back to the version 1 schema in its logs.

0 replies

thepont · 2019-11-19T22:53:53Z

thepont
Nov 19, 2019

So nested in the logs ( journalctl ) is the actual issue:

Nov 20 09:42:08 blue-pont dockerd[427]: time="2019-11-20T09:42:08.376878027+11:00" level=warning msg="failed to upload schema2 manifest: unauthorized: This operation would exceed the storage allotment for this account. - falling back to schema1"
Nov 20 09:42:08 blue-pont dockerd[427]: time="2019-11-20T09:42:08.376989057+11:00" level=warning msg="[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker.pkg.github.com registry NOW to avoid future disruption."
Nov 20 09:42:10 blue-pont dockerd[427]: time="2019-11-20T09:42:10.260794961+11:00" level=error msg="Not continuing with push after error: manifest invalid: Only schema version 2 is supported"

Seems our account is over the storage allocation ( I’m not the certain what this limit is ) and therefore we are no longer able to upload our images.

It is attempting to upload using schema 2 then failing and falling back, docker does not know how to handle the error from the github private repo and is therefor falling back in a reall confusing manner.

So i am not certain if we can delete older packages to free up space or what the limit is.

I used journalctl | grep dockerd (since I aways forget the flags) to get the full logs on Arch Linux, this will be different on OSX (https://docs.docker.com/docker-for-mac/troubleshoot/) or windows (https://docs.docker.com/v17.12/docker-for-windows/troubleshoot/).

But Docker will log the reason it falled back to the version 1 schema in its logs.

0 replies

aledalgrande · 2019-11-20T00:02:30Z

aledalgrande
Nov 20, 2019

We are seeing the same error and we have fixed tag length. Started after a few days, it was working before. Hopefully some staff member can write here to give more information.

0 replies

aledalgrande · 2019-11-20T00:21:28Z

aledalgrande
Nov 20, 2019

A related question: if the space is limited, can we automatically remove old images via some command/API?

0 replies

WyriHaximus · 2019-11-20T06:46:11Z

WyriHaximus
Nov 20, 2019
Author

Yes we can, I already have a work in progress action to remove package versions: https://github.com/WyriHaximus/github-action-delete-package

Which is based on this bit of documentation: https://help.github.com/en/github/managing-packages-with-github-packages/deleting-a-package#deleting-a-version-of-a-private-package

The thing is I’m trying to get the list of this packages with this GraphQL query but it’s giving me an unexpected not found error:

query {
  repository(owner:"OWNER", name:"REPOSITORY") {
    registryPackages(last:20) {
      edges {
        node {
          id,
          name
        }
      }
    }
  }
}

Glueing these two together and get them to work should make you able to delete old docker image tags.

0 replies

aledalgrande · 2019-11-20T18:24:50Z

aledalgrande
Nov 20, 2019

UPDATE: When we increased our total quota for the registry the action started working.

Also looking into that delete API, but we have a different approach because we don’t want to delete last n, we want to delete all images related to a tag too. Pretty simple to call the API once you get your head around it.

I find compounding actions with uses increases the total run time (have to download the image and run it), so unless the task is complex, it’s better to add a script/step to do it IMO. Also the least dependencies the better.

0 replies

thepont · 2019-11-20T21:42:51Z

thepont
Nov 20, 2019

I might be stating things that people already know, But talking to github support they advised that the default storage for Github team is:

GitHub Team

Storage: 2GB
Data Transfer: 10GB

Also that the organisation storage can be checked by viewing: https://github.com/organizations/ORG/settings/billing

The following works for me, to list versions of packages on a project.

curl -X POST \
    -H "Accept: application/vnd.github.package-deletes-preview+json" \
    -H "Authorization: bearer TOKEN" \
    -d '{"query":"query {repository(owner:\"ORG\", name:\"REPOSITORY\") {registryPackages(last:20) {edges{node{id, name, versions(last:20){edges {node {id}}}}}}}}"}' \
    https://api.github.com/graphql

0 replies

aledalgrande · 2019-11-21T00:31:59Z

aledalgrande
Nov 21, 2019

One thing you have to be careful with, we just discovered, is not deleting all the versions for a package; Github at that point will prevent you from publishing any version under that name and will return this error:

blob upload invalid: Docker image with name "<image name>" is already associated with another repository under owner "<account>"

Still waiting for an answer from support to understand how to fix it.

0 replies

WyriHaximus · 2019-11-22T07:16:16Z

WyriHaximus
Nov 22, 2019
Author

@thepont Yeah that fixes it. Would be nice to get a more descriptive error message in the long run tbh. Thank you for fixing the GraphQL query as well, they will get me to something working this weekend 🎉.

@aledalgrande That’s good to know! Wanted to do a latest tag for master builds and SHA or tag title for tagged releases, but sounds like the latest tag for master builds isn’t an option then :(. The reason I’m also doing an Action is to both give back and make it easier to compose actions between my projects. I’m intending to build an auto clean up of old image tags that aren’t relevant anymore but are still taking up space. (Also where I can I’m building Actions directly on alpine with a quick package install keeping that build time as low as possible.)

@thepont @aledalgrande Thank you both for all the help on this. I’ve accepted @thepont’s post because it’s a nice concise summary, but want to let you know you both been very helpful 👍.

0 replies

patped · 2019-12-19T09:51:31Z

patped
Dec 19, 2019

Any update from support on this error?

0 replies

saesh · 2020-01-10T16:41:51Z

saesh
Jan 10, 2020

Did you ever get an answer? I am having the exact same problem now.

0 replies

kevin-beatdapp · 2020-01-14T20:02:58Z

kevin-beatdapp
Jan 14, 2020

I’m experiencing the same issue now, were you able to find a solution?

0 replies

dieswaytoofast · 2020-01-18T00:55:19Z

dieswaytoofast
Jan 18, 2020

How did any of you delete a docker package?
when i try to do so, it sez. “Version deletion is currently unsupported for docker”

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Community

Pushing docker image to GPR: manifest invalid: Only schema version 2 is supported #25315

{{title}}

Replies: 19 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Pushing docker image to GPR: manifest invalid: Only schema version 2 is supported #25315

Replies: 19 comments

WyriHaximus Nov 16, 2019 Author

WyriHaximus Nov 19, 2019 Author

WyriHaximus Nov 20, 2019 Author

WyriHaximus Nov 22, 2019 Author

WyriHaximus
Nov 16, 2019
Author

WyriHaximus
Nov 19, 2019
Author

WyriHaximus
Nov 20, 2019
Author

WyriHaximus
Nov 22, 2019
Author