Action started to push manifest indexes instead of images for a single platform

[bubenkoff]

Troubleshooting

Behaviour

Steps to reproduce this issue

1. Set up build and push action with a single platform (non-multiarch build)

Expected behaviour

Image is pushed to the repository as an image, not manifest index

Actual behaviour

Image is pushed as a manifest index

[dongho-jung]

the first one works fine though the second one pushes wrong manifest indexes. it seems that buildkit version is culprit?

[dongho-jung]

I solved this by setting driver-opts in setup-buildx-action step

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          version: latest
          endpoint: builders
          driver-opts: 'image=moby/buildkit:v0.10.5'

[jedevc]

See my comment here:

This is the new provenance feature, which is enabled by default - you can get the contents using buildx imagetools:

$ docker buildx imagetools inspect <your-image> --format '{{ json .Provenance }}'

To disable it on build, you can explicitly set --provenance=false. You can find out more in the docs here.

Is there an issue with the single platform being uploaded as an index instead of a manifest? Is there a specific tool you're using that's giving issues?

[martadinata666]

so in simple terms add --provenance=false to every docker buildx build in CI? 🤔

[jedevc]

@martadinata666 that's definitely a solution you can try 🎉

But yeah, not ideal - the idea for enabling provenance by default is to enable better introspection of built images, where they come from, how they were built, etc. We want to make it easier by default to understand where an image has come from, to make it easier to decide how and whether to trust random images on the internet. 🤞

Is there a reason that it matters whether buildkit outputs an index instead of an image manifest here? I'm not able to think of a use case where this should matter, except maybe for the known bug with docker manifest: docker/buildx#1509. Ideally, how the image is stored in the registry shouldn't be a detail that end-users need to be aware of - if you are, I'm interested in why!

In docker/buildx#1509 (comment), I give some workarounds for the docker manifest issue - using imagetools should be a drop-in replacement for docker manifest that properly handles OCI indexes. If that's not an acceptable workaround, is there a reason why, so we can work on that?

[Legomegger]

I solved this by setting driver-opts in setup-buildx-action step

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          version: latest
          endpoint: builders
          driver-opts: 'image=moby/buildkit:v0.10.5'

tried this and got
ERROR: failed to initialize builder builder-416a51ae-43b3-43a2-b551-c9b764437ef9 (builder-416a51ae-43b3-43a2-b551-c9b764437ef90): error during connect: Get "http://builders:2375/v1.24/containers/buildx_buildkit_builder-416a51ae-43b3-43a2-b551-c9b764437ef90/json": dial tcp: lookup builders on 127.0.0.53:53: server misbehaving

[jedevc]

@Legomegger you'll need to set endpoint appropriately for your build - it's optional, so if you're not doing anything special with it, you should be able to just drop that field.

[bubenkoff]

@jedevc thanks a lot for solving this mystery!
so is there a away to set --provenance=false, do you have an example?
pinning driver-opts: 'image=moby/buildkit:v0.10.5' seem to be quite extreme as it's exact version and might break again later on?

[nanake]

@jedevc thanks a lot for solving this mystery! so is there a away to set --provenance=false, do you have an example? pinning driver-opts: 'image=moby/buildkit:v0.10.5' seem to be quite extreme as it's exact version and might break again later on?

Not now, just wait for #746 to land to use provenance: false.

[bubenkoff]

@jedevc
and on the reasons why this is important so that the previous behavior stays: building multiarch images using build is very slow due to virtualization, hence the need of parallel build using different arch runners that push separate images, then use https://github.com/Noelware/docker-manifest-action to push the manifest index using those separate images. This way it's like 10x faster.

[martadinata666]

docker actions wise, i personally set to specific version right now.

  - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          version: v0.9.1

[jedevc]

@bubenkoff yup, buildx actually can do parallel native builds without needing to do manual manifest manipulation - however, it's definitely not as well presented as it could be. We're tracking that here: docker/buildx#1510. If there's something you think we should cover in relation to the topic, please do mention it in that thread 🎉

[Vaub]

Is there a reason that it matters whether buildkit outputs an index instead of an image manifest here? I'm not able to think of a use case where this should matter, except maybe for the known bug with docker manifest: docker/buildx#1509. Ideally, how the image is stored in the registry shouldn't be a detail that end-users need to be aware of - if you are, I'm interested in why!

If it helps, it broke our Google Cloud Run deployments, they don't seem to support this new format under-the-hood (at least in our region it causes an image X not found error). There isn't a lot of documentation from how they manage images, but I feel it is related to this bug. In any case, we reverted to v0.9.1 for now.

[bekriebel]

Is there a reason that it matters whether buildkit outputs an index instead of an image manifest here? I'm not able to think of a use case where this should matter, except maybe for the known bug with docker manifest: docker/buildx#1509. Ideally, how the image is stored in the registry shouldn't be a detail that end-users need to be aware of - if you are, I'm interested in why!

If it helps, it broke our Google Cloud Run deployments, they don't seem to support this new format under-the-hood (at least in our region it causes an image X not found error). There isn't a lot of documentation from how they manage images, but I feel it is related to this bug. In any case, we reverted to v0.9.1 for now.

This also broke our Cloud Run deployments that are pulling from Artifact Registry. It manifests as Cloud Run saying "Image '' not found". It is fixed by adding provenance: false to my docker/build-push-action@v3 with values.

[jakeonfire]

with the update to buildx v0.10.0 it started using manifest v1?!

from the Manifest output:

    "containerimage.descriptor": {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:3a1b902d20d975b7a38a36ccc8cbeba629599501fd2bbd6a0414678580a601c5",
      "size": 856
    },

and that causes issues later on (during docker pull which doesn't support v1).

for comparison it used to look like this:

    "containerimage.descriptor": {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:7e0951e63d2cfaba5486fd9e899d4712eca10b23c0f49c46fdd253fc01dc36c6",
      "size": 5817,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },

@bekriebel's solution restored the previous behavior:

It is fixed by adding provenance: false to my docker/build-push-action@v3

[joestringer]

I've also hit issues with this, however setting provenance: false is not working for me (see cilium/cilium#23206 (comment)). However, pinning the version does appear to work correctly, so I've hardcoded the v0.9.1 buildx version for now in order to mitigate this issue. This is with docker/build-push-action@37abcedcc1da61a57767b7588cb9d03eb57e28b3 (should be v3.3.0).

EDIT: It looks like I'm also hitting this issue with docker/build-push-action@c56af957549030174b10d6867f20e78cfd7debc5 elsewhere, which suggests that the issue may lie more with the docker buildx v0.10.0 release rather than this docker action.

EDIT2: Note that the version setting is for the docker/setup-buildx-action, not docker/build-push-action.

[mgmarino]

Just adding that this also broke deployments to AWS lambda, which currently does not support multi-architecture containers. We were confronted with the following error:

Resource handler returned message: "The image manifest or layer media type for the source image xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/lambda/container:vXXX is not supported

We noticed that the new builds (with buildx 0.10.0 instead of 0.9) added the following flags by default:

--provenance mode=min,inline-only=true,builder-id=https://github.com/org/repo/actions/runs/xxxxxx

We were able to solve this by setting provenance: false.

IMHO it was not ideal to have this breaking change introduced without e.g. a major version bump. Going forward, should we expect breaking changes like this, i.e. should we rather pin to minor versions?

Thanks!

[jedevc]

@Vaub @bekriebel @mgmarino I've done some investigation and opened a separate tracking issue for the GCR/Lambda failures: docker/buildx#1533. This is related to those platforms lack of support for multi-platform images - I've also given some other temporary workarounds in that thread.

---

@jakeonfire the update to buildx causes the format of the output to upgrade from docker manifest v2 to oci v1. See the mediatypes:

• On Buildx v0.9 : application/vnd.docker.distribution.manifest.v2+json
• On Buildx v0.10: application/vnd.oci.image.index.v1+json

See https://github.com/opencontainers/image-spec for more info.

[bekriebel]

@Vaub @bekriebel @mgmarino I've done some investigation and opened a separate tracking issue for the GCR/Lambda failures: docker/buildx#1533. This is related to those platforms lack of support for multi-platform images - I've also given some other temporary workarounds in that thread.

There's lack of support on the runners, but also a seeming lack of support on the container registries. In both Google Artifact Registry and GitHub Container registry, an image pushed without provenance: false is appearing as three separate images. One of them can be pulled properly if you point to it, but the one that gets tagged appears to be just a manifest entry and can't be pulled by the services. Personally, I'll just be keeping provenance disabled until this format is better supported by existing cloud infrastructures.

[devunt]

This also causing ArgoCD Image Updater to fails with Amazon ECR registry. It seems they're not supporting the new format.

[alexellis]

@jedevc it seems like this breaks our tutorial for building two native images and compiling them into a manifest.

https://actuated.dev/blog/how-to-run-multi-arch-builds-natively

We can tell users to set provenance: false when using the v4 action, but is there a way to use these new provenance-enabled manifests to create a multi-arch image / manifest as per the blog post?

Or is that no-longer going to be possible going forward if using provenance?

[jedevc]

@alexellis docker manifest unfortunately doesn't work with multi-platform indexes as inputs. In an above message, I link to a list of workarounds: docker/buildx#1509 (comment).

I'd personally recommend using buildx imagetools which supports OCI/docker media types, merging manifest lists, and all of that.

[mqasimsarfraz]

I'd personally recommend using buildx imagetools which supports OCI/docker media types, merging manifest lists, and all of that.

@jedevc I have been using buildx imagetools to create a manifest lists with OCI media types but struggling to find a way to add OCI annotations to make GitHub happy with descriptions and all ? I am wondering if that is something imagetool even supports (or should support)?

[seebeen]

@jedevc it seems like this breaks our tutorial for building two native images and compiling them into a manifest.

https://actuated.dev/blog/how-to-run-multi-arch-builds-natively

We can tell users to set provenance: false when using the v4 action, but is there a way to use these new provenance-enabled manifests to create a multi-arch image / manifest as per the blog post?

Or is that no-longer going to be possible going forward if using provenance?

Did you get an answer?

[mqasimsarfraz]

@jedevc it seems like this breaks our tutorial for building two native images and compiling them into a manifest.
https://actuated.dev/blog/how-to-run-multi-arch-builds-natively
We can tell users to set provenance: false when using the v4 action, but is there a way to use these new provenance-enabled manifests to create a multi-arch image / manifest as per the blog post?
Or is that no-longer going to be possible going forward if using provenance?

Did you get an answer?

@seebeen As mentioned in the comment here using buildx imagetools should do the trick. The complete steps are listed in "Distribute build across multiple runners". More examples (index pushing, index merging)

[seebeen]

That doesnt work. No annotations get applied. It doesn't matter where you add them.
I've compiled a custom manifest file myself, using sha's from previous release, and I'm overwriting the manifest -> no show 😢

[mqasimsarfraz]

Can you please share the steps you are trying so I can get an idea? Also, in case of annotations if you want to add them to final index then you must use buildx imagetools create --annotation .... The support for --annotation flag was recently merged so be sure to use up to date version:

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v2
    with:
    # TODO: Remove once buildx v0.12.0 is released with https://github.com/docker/buildx/pull/1965
    version: https://github.com/docker/buildx.git#e5cee892ed4fa2e0af53c54c09131c96c68383ff

We are using them here and works fine.

[seebeen]

Here is the entire workflow file

name: Docker builds
on:
  release:
    types: [published]

permissions:
  contents: read
  packages: write

env:
  REGISTRY_IMAGE: ghcr.io/oblakstudio/woosync
  BUILDX_NO_DEFAULT_ATTESTATIONS: 1

jobs:
  build:
    name: Build image
    strategy:
      fail-fast: false
      matrix:
        platform:
          - linux/amd64
          - linux/arm64
        include:
          - platform: linux/amd64
            runner: amd64-builder
          - platform: linux/arm64
            runner: arm64-builder

    runs-on: ${{ matrix.runner }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Import GPG key
        uses: crazy-max/ghaction-import-gpg@v5
        with:
          gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
          passphrase: ${{ secrets.GPG_PASSPHRASE }}
          git_config_global: true
          git_user_signingkey: true
          git_commit_gpgsign: true

      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to Docker Hub
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: oblakbot
          password: ${{ secrets.OBLAK_BOT_TOKEN }}

      - name: Docker Metadata
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY_IMAGE }}

      - name: Build Docker Image
        id: build
        uses: docker/build-push-action@v4
        with:
          context: ./
          platforms: ${{ matrix.platform }}
          push: ${{ github.event_name != 'pull_request' }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha,scope=${{ matrix.platform }}
          cache-to: type=gha,mode=max,scope=${{ matrix.platform }}
          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true,annotation.org.opencontainers.image.title=WooCommerce Sync Service v2,annotation.org.opencontainers.image.authors=Oblak Studio <support@oblak.studio>,annotation.org.opencontainers.image.source=https://github.com/oblakstudio/wcs,annotation.org.opencontainers.image.description=Docker Optimized WooSync v2,annotation.org.opencontainers.image.licenses=MIT
          provenance: false

      - name: Export Image Digest
        run: |
          mkdir -p /tmp/build-digest
          digest="${{ steps.build.outputs.digest }}"
          digest="${digest#sha256:}"
          touch "/tmp/build-digest/$digest"
        shell: bash

      - name: Upload digest
        uses: actions/upload-artifact@v3
        with:
          name: digests
          path: /tmp/build-digest/*
          if-no-files-found: error
          retention-days: 1

      - name: Clear temporary directory
        run: |
          rm -rf /tmp/build-digest/*
        shell: bash

  merge:
    name: Merge manifest
    runs-on: ubuntu-latest
    needs:
      - build
    steps:
      - name: Download digests
        uses: actions/download-artifact@v3
        with:
          name: digests
          path: /tmp/build-digest

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY_IMAGE }}
          tags: |
            type=ref,event=branch
            type=ref,event=pr
            type=sha
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=semver,pattern={{major}}
          labels: |
            org.opencontainers.image.source="https://github.com/oblakstudio/wcs"
            org.opencontainers.image.authors="Oblak Studio <support@oblak.studio>"
            org.opencontainers.image.title="WooCommerce Sync Service v2"
            org.opencontainers.image.description="Docker Optimized WooSync v2"
            org.opencontainers.image.licenses="MIT"

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: oblakbot
          password: ${{ secrets.OBLAK_BOT_TOKEN }}

      - name: Create manifest list and push
        working-directory: /tmp/build-digest
        run: |
          docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
            $(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)

      - name: Inspect image
        run: |
          docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}
      - name: Delete artifact
        uses: geekyeggo/delete-artifact@v2
        with:
          name: my-artifact
          failOnError: false
          ```

[mqasimsarfraz]

The workflow you shared will only add the annotations to the individual manifests and not the index (refereed by the tag). You can try following merge step to see if it helps:

  merge:
    name: Merge manifest
    runs-on: ubuntu-latest
    needs:
      - build
    steps:
      - name: Download digests
        uses: actions/download-artifact@v3
        with:
          name: digests
          path: /tmp/build-digest

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          # TODO: Remove once buildx v0.12.0 is released with https://github.com/docker/buildx/pull/1965
          version: https://github.com/docker/buildx.git#e5cee892ed4fa2e0af53c54c09131c96c68383ff

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY_IMAGE }}
          tags: |
            type=ref,event=branch
            type=ref,event=pr
            type=sha
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=semver,pattern={{major}}
          labels: |
            org.opencontainers.image.source="https://github.com/oblakstudio/wcs"
            org.opencontainers.image.authors="Oblak Studio <support@oblak.studio>"
            org.opencontainers.image.title="WooCommerce Sync Service v2"
            org.opencontainers.image.description="Docker Optimized WooSync v2"
            org.opencontainers.image.licenses="MIT"

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: oblakbot
          password: ${{ secrets.OBLAK_BOT_TOKEN }}

      - name: Create manifest list and push
        working-directory: /tmp/build-digest
        run: |
          docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
            $(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *) \
            --annotation index:org.opencontainers.image.title=WooCommerce \
            --annotation index:org.opencontainers.image.title="WooCommerce Sync Service v2"
      - name: Inspect image
        run: |
          docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}
      - name: Delete artifact
        uses: geekyeggo/delete-artifact@v2
        with:
          name: my-artifact
          failOnError: false
          ```

(Also, can remove annotations from Build Docker Image)