GitHub Action produces unknown architecture and OS

[mu88]

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

Behaviour

Steps to reproduce this issue

1. Run this GitHub Action
2. Pull the created Docker image onto a Raspberry Pi via docker pull mu88/thisisyourlife:latest

Expected behaviour

The image can be pulled.

Actual behaviour

latest: Pulling from mu88/shopandeat
no matching manifest for linux/arm64/v8 in the manifest list entries

When running hub-tool tag ls mu88/thisisyourlife --format json on my Win11 machine, it shows this:

[
  {
    "Name": "mu88/thisisyourlife:latest",
    "FullSize": 102816471,
    "LastUpdated": "2023-02-12T09:48:05.871245Z",
    "LastUpdaterUserName": "mu88",
    "Images": [
      {
        "Digest": "sha256:a372572eb3b089293ff8334f4f9d70e6dd6900f48314c372d86a83f4ba9f0c5f",
        "Architecture": "arm64",
        "Os": "linux",
        "Variant": "v8",
        "Size": 102816471,
        "LastPulled": "2023-02-12T09:48:06.485083Z",
        "LastPushed": "2023-02-12T09:48:05.470786Z",
        "Status": "active"
      },
      {
        "Digest": "sha256:85ebb19d1ec6022c5d0999c80dd58d847497a4afc35bcecacf854f91e7226560",
        "Architecture": "unknown",
        "Os": "unknown",
        "Variant": "",
        "Size": 19940,
        "LastPulled": "2023-02-12T09:48:06.520255Z",
        "LastPushed": "2023-02-12T09:48:05.636699Z",
        "Status": "active"
      }
    ],
    "LastPulled": "2023-02-12T09:49:00.217342Z",
    "LastPushed": "2023-02-12T09:48:05.871245Z",
    "Status": "active"
  },
  {
    "Name": "mu88/thisisyourlife:2.2",
    "FullSize": 102816471,
    "LastUpdated": "2023-02-12T09:48:06.748996Z",
    "LastUpdaterUserName": "mu88",
    "Images": [
      {
        "Digest": "sha256:a372572eb3b089293ff8334f4f9d70e6dd6900f48314c372d86a83f4ba9f0c5f",
        "Architecture": "arm64",
        "Os": "linux",
        "Variant": "v8",
        "Size": 102816471,
        "LastPulled": "2023-02-12T09:48:06.485083Z",
        "LastPushed": "2023-02-12T09:48:05.470786Z",
        "Status": "active"
      },
      {
        "Digest": "sha256:85ebb19d1ec6022c5d0999c80dd58d847497a4afc35bcecacf854f91e7226560",
        "Architecture": "unknown",
        "Os": "unknown",
        "Variant": "",
        "Size": 19940,
        "LastPulled": "2023-02-12T09:48:06.520255Z",
        "LastPushed": "2023-02-12T09:48:05.636699Z",
        "Status": "active"
      }
    ],
    "LastPulled": "2023-02-12T09:49:00.217342Z",
    "LastPushed": "2023-02-12T09:48:06.748996Z",
    "Status": "active"
  },
...

I've already created this issue in a Docker forum. I first had the problem with this repo resp. the Docker image mu88/shopandeat:latest. Then I thought that I can resolve it by deleting all tags from the Docker Hub and building the image again, but now the problem is back for another repo and deleting the tags doesn't solve it.

Since I cannot reproduce this issue when running the GitHub Action locally (via act) or simply via docker build, I assume it's a problem of this GitHub Action step docker/build-push-action@v4.

Configuration

• Repository URL (if public): https://github.com/mu88/ThisIsYourLife
• Build URL (if public):

name: Release

on:
  push:
    # Sequence of patterns matched against refs/tags
    tags:
      - '*' # Push events to matching everything, e. g. "v1" or "2021.2.2.1"

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Setup .NET
      uses: actions/setup-dotnet@v3
      with:
        dotnet-version: '7.0'
    - name: Restore dependencies
      run: dotnet restore ThisIsYourLife.sln
    - name: Build
      run: dotnet build --no-restore ThisIsYourLife.sln
    - name: Test
      run: dotnet test --no-build --verbosity normal ThisIsYourLife.sln
    - name: Set up QEMU
      uses: docker/setup-qemu-action@v2
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2
    - name: Login to DockerHub
      uses: docker/login-action@v2
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}
    - name: Build and push Docker image for Raspberry Pi
      id: docker_build
      uses: docker/build-push-action@v4
      with:
        push: true
        tags: |
          mu88/thisisyourlife:latest
          mu88/thisisyourlife:${{ github.ref_name }}
        file: src/WebApp/Dockerfile
    - name: Build single-file EXE
      run: dotnet publish src/WebApp/WebApp.csproj -r win-x64 -p:PublishSingleFile=true -p:Configuration=Release --self-contained true
    - name: Create ZIP for release
      uses: thedoctor0/zip-release@master
      with:
        type: 'zip'
        filename: ThisIsYourLife.${{ github.ref_name }}.zip
        directory: 'src/WebApp/bin/Release/net7.0/win-x64/publish'
    - name: Create CHANGELOG
      id: changelog
      uses: Requarks/changelog-action@v1
      with:
        token: ${{ github.token }}
        tag: ${{ github.ref_name }}
        excludeTypes: build,docs,other,style,chore(deps)
    - name: Create GitHub Release
      uses: softprops/action-gh-release@v1
      with:
        files: src/WebApp/bin/Release/net7.0/win-x64/publish/ThisIsYourLife*.zip
        tag_name: ${{ github.ref_name }}
        body: ${{ steps.changelog.outputs.changes }}

Logs

Download the log file of your build and attach it to this issue.

[jedevc]

What version of Docker is running on the raspberry pi? You can get this information from docker info. I think the version of Docker is probably quite an old version, and isn't able to pull OCI images - which is the default in the latest versions of buildx (which the build-push-action uses to build images).

The extra manifest is attached as part of the new provenance feature - see the release notes for v4.0.0 for more information: https://github.com/docker/build-push-action/releases/tag/v4.0.0

[mu88]

Here's the output of dotnet info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 10
  Running: 10
  Paused: 0
  Stopped: 0
 Images: 17
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.84-v8+
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.705GiB
 Name: <<name of my Raspi>>
 ID: IDOS:QQKL:UAX6:G6LD:FEEG:QVRA:HKCA:W5RD:3VOW:EV4N:XJJJ:O2E6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: mu88
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support

Doesn't seem that old to me, doesn't it :)

[crazy-max]

Doesn't seem you're building for your platform in your workflow. Atm it builds a linux/amd64 image which is the native platform on ubuntu-latest GitHub Runner:

$ docker buildx imagetools inspect mu88/thisisyourlife:latest
Name:      docker.io/mu88/thisisyourlife:latest
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:023882a6a787ed0ec0d7922cd29a9d81041f2b6d22bfd6d9360ed45a20065b0d

Manifests:
  Name:        docker.io/mu88/thisisyourlife:latest@sha256:a372572eb3b089293ff8334f4f9d70e6dd6900f48314c372d86a83f4ba9f0c5f
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64

  Name:        docker.io/mu88/thisisyourlife:latest@sha256:85ebb19d1ec6022c5d0999c80dd58d847497a4afc35bcecacf854f91e7226560
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.type:   attestation-manifest
    vnd.docker.reference.digest: sha256:a372572eb3b089293ff8334f4f9d70e6dd6900f48314c372d86a83f4ba9f0c5f

Looks like you're missing the platforms input with linux/arm64:

    - name: Build and push Docker image for Raspberry Pi
      id: docker_build
      uses: docker/build-push-action@v4
      with:
        push: true
        platforms: linux/amd64,linux/arm64
        tags: |
          mu88/thisisyourlife:latest
          mu88/thisisyourlife:${{ github.ref_name }}
        file: src/WebApp/Dockerfile

More info: https://docs.docker.com/build/ci/github-actions/multi-platform/

[mu88]

Specifying the platforms parameter results in further issues with building the .NET code (see this issue).

AFAIK specifying mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim-arm64v8 as base image (see here) should be sufficient - at least it has been working this way for years.

[mu88]

@crazy-max what would be the syntax for setting --provenance=false in my GitHub Action? Like this:

    - name: Build and push Docker image for Raspberry Pi
      id: docker_build
      uses: docker/build-push-action@v4
      with:
        push: true
        provenance: false
        tags: |
          mu88/thisisyourlife:latest
          mu88/thisisyourlife:${{ github.ref_name }}

[mu88]

Okay, using provenance: false works 👍🏻

[shanduur]

Got the same issue in my GitHub action on test repo, with minimal config: https://github.com/shanduur/template-ci-container/pkgs/container/template-ci-container/76745715?tag=v1.0.0-beta.1-alt

[nwhobart]

Got the same issue in my GitHub action on test repo, with minimal config: https://github.com/shanduur/template-ci-container/pkgs/container/template-ci-container/76745715?tag=v1.0.0-beta.1-alt

@shanduur I set provenance: false in my workflow as well and it worked. I'll be damned if I know what that means. :)

[mu88]

@crazy-max what's the future of this issue? I'd like to remove provenance: false sooner or later... is there something you can do or do we have to wait for cross-rs/cross#1214?

[shanduur]

@mu88 @nwhobart looks like the ghcr.io expects the images to be in OCI format, and Docker format is not compatible with it due to the additional metadata. Setting provenance: false disables adding those additional Docker metadata, and the image is more compliant with OCI spec.

[mu88]

Just out of curiosity 🙂 could you elaborate a bit more on why a container registry that is not fully compatible with all Docker features is used? From a customer/developer perspective, that's a bit hard to understand, to be honest, but I'm pretty sure there are excellent reasons.

[jedevc]

The root cause to me seems to be with cross-compilation - building an image for platform A on platform B. Setting --platform A means you get the wrong metadata in the image output (which because images are now by default multi-platform, causes messages like no matching manifest for linux/arm64/v8 in the manifest list entries), while setting --platform B means you invoke QEMU by default (which is slower and may have compatibility issues).

The fix for this is to use the docker build helpers for cross-compilation, and not rely on the assumption that machines of arch A will consume incorrect metadata for arch B. See the multi-platform guide for a more detailed introduction, but to summarize:

• Use FROM --platform=$BUILDPLATFORM in your stages to explicitly have that stage use the architecture of the build machine, regardless of what the cli --platform arg is set to. This ensures that the build is native, and that the resulting metadata is correct.
• Use FROM --platform=$TARGETPLATFORM in your stages to explicitly have that stage use the architecture of the target machine, regardless of what the cli --platform arg is set to.

If you are building multi-platform images, you should make sure that --platform is set to your target platform - otherwise the metadata will be incorrect.

---

@mu88:

I'd like to remove provenance: false sooner or later... is there something you can do or do we have to wait for cross-rs/cross#1214?

A few options:

1. Revert to @V3 of this action

2. Set provenance: false

3. Wait for "no matching manifest" on cross v0.2.5 cross-rs/cross#1214. The TL;DR of the issue boils down to:

   docker run and docker pull, when a manifest is a single platform, will try to use that platform, if it's multi-arch, it'll use the same platform as the default, and if it doesn't exist panic.

   This behaviour is consistent across all tools in the container ecosystem, and is actually implemented in containerd. Multi-platform images have been supported by most container tooling for some time now, and the provenance change has found some places which don't support this unfortunately. See above for the correct usage of --platform and $BUILDPLATFORM.

@shanduur:

looks like the ghcr.io expects the images to be in OCI format, and Docker format is not compatible with it due to the additional metadata. Setting provenance: false disables adding those additional Docker metadata, and the image is more compliant with OCI spec.

This isn't true. GHCR supports images with both Docker media types and OCI media-types, like most registries today, including Docker Hub. When provenance is generated, the default from v4 of this action (and buildx v0.10), the built image is actually an OCI image, compared to previous versions where it was using Docker media-types. There is no additional out-of-spec metadata included by docker with provenance enabled, the images we generate are entirely in-spec with OCI.

[mu88]

Cool, thank you very much for your in-depth explanation 💪🏻 setting FROM --platform=$BUILDPLATFORM seems to be the way to go for the .NET guys (see here).

[shanduur]

@jedevc thanks for clarification!