4.0.0 + --provenance false fails with ERROR: attestations are not supported by the current buildkitd

[polarathene]

Behaviour

• Updated to 4.0.0 and explicitly opted out of provenance.
• A workflow with a matrix run has 3 out of 4 runners failing with the only difference seeming to be buildx version 0.10.0, while the passing runner had buildx version 0.10.2.

Presumably this is a transient issue with Github. I've re-run the failed jobs twice but they continue to use the older buildx version and failing:

Docker Engine: 20.10.22
Buildx: 0.10.0

/usr/bin/docker buildx build --cache-from type=local,src=/tmp/.buildx-cache --iidfile /tmp/docker-build-push-AubJbU/iidfile --platform linux/amd64 --provenance false --tag mailserver-testing:ci --load --metadata-file /tmp/docker-build-push-AubJbU/metadata-file .

ERROR: attestations are not supported by the current buildkitd
Error: buildx failed with: ERROR: attestations are not supported by the current buildkitd

This is bizarre since we've clearly opted out of provenance with the CLI option from this action?

Steps to reproduce this issue

Probably not easy to reproduce, but here is our affected workflow:

jobs:
  run-tests:
    name: 'Test'
    runs-on: ubuntu-22.04
    strategy:
      matrix:
        part: [serial, parallel/set1, parallel/set2, parallel/set3]
      fail-fast: false
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          # Required to retrieve bats (core + extras):
          submodules: recursive

      # Get the cached build layers from the build job:
      # This should always be a cache-hit, thus `restore-keys` fallback is not used.
      # No new cache uploads should ever happen for this job.
      - name: 'Retrieve image built from build cache'
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: cache-buildx-${{ inputs.cache-key }}

      # Importing from the cache should create the image within approx 30 seconds:
      # Earlier `buildx` + `qemu` steps are not needed as no cache is exported,
      # and only a single platform (AMD64) is loaded:
      - name: 'Build AMD64 image from cache'
        uses: docker/build-push-action@v4.0.0
        with:
          context: .
          tags: mailserver-testing:ci
          # Export the built image to the Docker host for use with BATS:
          load: true
          # Rebuilds the AMD64 image from the cache:
          platforms: linux/amd64
          cache-from: type=local,src=/tmp/.buildx-cache
          # Disable provenance attestation: https://docs.docker.com/build/attestations/slsa-provenance/
          provenance: false

In the past we have not needed the buildx action for this workflow (a previous workflow it is dependent on provides a build and exports / uploads the image as a cache). Both the passing workflow (buildx 0.10.2) and failing workflows (buildx 0.10.0) successfully import the same cache. Nothing else appears to be different in the action logs.

Expected behaviour

Workflow completes without error about missing attestations support.

Actual behaviour

Workflow fails, despite opting out of attestations with provenance: false.

Configuration

• See "Steps to reproduce this issue" section above.
• Build URL / logs are linked to in the affected PR.

Logs

logs_8988.zip

[polarathene]

Raised issue upstream on buildx: docker/buildx#1608

Failed attempt to resolve

Observed in their releases for 0.10.1 that existing images with attestations may not be compatible with buildx --load, and since we upgraded the action from v3.3 to v4, I suspect our image cache hit is running into that problem with buildx 0.10.0?

---

UPDATE: Cleared entire CI cache and performed a full workflow re-run. No difference, still getting the same failure.

---

I think for now, I must change to provenance: true since that was the default in docker/build-push-action v3.3? In a week or so, Github CI should have Docker Engine v23 available (updating bundled BuildKit), and buildx should be 0.10.2. Once that happens provenance: false will probably work without the weird error from trying to opt-out.

Alternatively, adding docker/setup-buildx-action into this workflow to use the docker-container driver (and thus newer BuildKit) is probably another option? Even though we're only loading the retrieved build cache export from a previous build workflow, so that we can run the container locally for tests (BuildKit is still relevant?).

[polarathene]

I am fairly certain I understand it now 👍

• Affected workflow used docker driver instead of docker-container.
• CI Docker Engine bundles BuildKit release from 2 years ago (soon to be resolved with v23 update).
• docker/build-push-action appends --provenance option to buildx command if explicitly configured in the workflow (or compatible BuildKit version detected).
• buildx 0.10.0 would fail early on attestation support with old BuildKit, before acknowledging opt-out with --provenance false. 0.10.1 or 0.10.2 presumably fixed that mishap to only fail when --provenance true (default, but presumably only when the BuildKit version supported it, hence no failure prior).

So, the only thing that changed was explicitly opting out with provenance: false in the action, added an explicit --provenance false option to buildx, mixed with bad timing of Github dependencies and the one workflow not using docker-container driver for newer BuildKit version 😅 (all while about to become a non-issue with Github as the CI in about a week)

Closing as I'm fairly confident in the troubleshooting and resolution 😎