Getting "no such manifest" since provenance was added

[ddelange]

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

Behaviour

• workflow file in the left column
• recently added by default: --provenance mode=max,builder-id=https://github.com/***/actions-runner-controller-releases/actions/runs/3939568621
• the manifest now seems broken

  $ docker manifest inspect ddelange/actions-runner:latest
  no such manifest: docker.io/ddelange/actions-runner:latest
  $ docker manifest inspect ghcr.io/ddelange/actions-runner-controller-releases/actions-runner:latest
  no such manifest: ghcr.io/ddelange/actions-runner-controller-releases/actions-runner:latest

• it also causes a third OS/Arch to show up on ghcr.io (on hub.docker.com it is hidden) with sha corresponding to exporting attestation manifest ...
  
• when I set provenance: false (I'm running v3.3.0), the manifests are fixed

Steps to reproduce this issue

workflow linked above

Expected behaviour

i expect the manifests on docker.io and ghcr.io not to be broken (or it should be a v4 bump or so and fixed differently? should I update docker cli?)

Actual behaviour

manifests broken, see above

Configuration

• Repository URL (if public): https://github.com/ddelange/actions-runner-controller-releases
• Build URL (if public): https://github.com/ddelange/actions-runner-controller-releases/actions/runs/3939568621/jobs/6739577887#step:7:134
• YAML https://github.com/ddelange/actions-runner-controller-releases/blob/b4691d61c3d4a5d8a05cff2e4c2ed2c9e8afaf96/.github/workflows/release-runners.yaml

Logs

logs_14.zip

[crazy-max]

This is expected. See docker/buildx#1509 (comment) and also docker/buildx#1509 (comment) for more context.

[ddelange]

Hi @crazy-max 👋 thanks for the quick reply!

do I understand correctly this change ships with buildx 0.10.0 (included in docker desktop 4.16.0)?

will docker manifest inspect work again if I were able to upgrade Docker Deskop? (need to upgrade MacOS to v11+ first...)

[ddelange]

looks like docker buildx imagetools inspect misses a verbosity flag, so I can't use it as a replacement for docker manifest inspect -v to sum up layer sizes: https://stackoverflow.com/a/73108928/5511061

[crazy-max]

do I understand correctly this change ships with buildx 0.10.0 (included in docker desktop 4.16.0)?

Not just Buildx but also BuildKit 0.11 (which is used by default with the setup-buildx-action step in your workflow (here) that creates a docker-container builder)

On Docker Desktop, the default builder is the docker driver (BuildKit in the Docker engine) which is still BuildKit 0.8 so there is no attestation support.

will docker manifest inspect work again if I were able to upgrade Docker Deskop? (need to upgrade MacOS to v11+ first...)

This is currently discussed in docker/buildx#1509. If you can add your use case with docker manifest cmd on this issue, that would help.

[billinghamj]

@crazy-max Could I suggest a pinned issue in this repo, to help people find the buildx issue?