header key "followpaths" contains value with non-printable ASCII characters

[andmos]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

👋
We're building an application via multistage Dockerfile:

FROM node:14-alpine AS frontend-build
WORKDIR /app

COPY ./frontend/package.json ./
COPY ./frontend/yarn.lock ./
RUN yarn install --frozen-lockfile

COPY ./frontend ./
RUN yarn build

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS dotnet-build
ARG GITHUB_TOKEN
WORKDIR /app
COPY ./backend/Directory.Build.props ./
COPY ./backend/Agenter/Epost/Epost.csproj ./Agenter/Epost/
COPY ./backend/Agenter/GoogleCloudPlatform/GoogleCloudPlatform.csproj ./Agenter/GoogleCloudPlatform/
COPY ./backend/Agenter/Intelecom/Intelecom.csproj ./Agenter/Intelecom/
COPY ./backend/Agenter/Rutesøk/Rutesøk.csproj ./Agenter/Rutesøk/
COPY ./backend/Data/Data.csproj ./Data/
COPY ./backend/Domene/Domene.csproj ./Domene/
COPY ./backend/EksterntApi/EksterntApi.csproj ./EksterntApi/
COPY ./backend/Identity/Identity.csproj ./Identity/
COPY ./backend/Import/Import.csproj ./Import/
COPY ./backend/Logikk/Logikk.csproj ./Logikk/
COPY ./backend/Web/Web.csproj ./Web/

RUN dotnet restore Web/Web.csproj

COPY ./backend ./
COPY --from=frontend-build /app/dist ./Web/wwwroot
RUN dotnet publish Web/Web.csproj -c Release -o out


FROM mcr.microsoft.com/dotnet/aspnet:7.0
WORKDIR /app
RUN apt update && apt install -y libc-dev libgdiplus
COPY --from=dotnet-build /app/out .
ENTRYPOINT ["dotnet", "Application.Reserve.Web.dll", "--urls", "http://0.0.0.0:8080"]

Via docker/build-push-action@v5:

      - name: Build and push docker image to gcr
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: eu.gcr.io/application/application:latest
          platforms: linux/amd64
          build-args: |
            GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}

For deployment on Google Cloud App Engine.

We picked up the project after some months, and the build now fails with

ERROR: failed to solve: Internal: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters
Error: buildx failed with: ERROR: failed to solve: Internal: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters

Our codebase has filenames containing the Norwegian æ ø å, but that has not been a problem before and is not a problem locally. A search for this error returns a fixed BuildKit issue from june 2023: moby/buildkit#3927

I've tried to work around the issue by using buildx which builds the image without error:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

But the problem then is that Google Cloud App doesn't support(?):

ERROR: (gcloud.app.deploy) Error Response: [3] Docker image eu.gcr.io/application/application:latest was either not found, or is not in Docker V2 format.

The image is present in the registry, but the container format is changed:

Image type
    OCI Image Index
Media type
    application/vnd.oci.image.index.v1+json

While the previous working image built without buildx had the following:

Image type
    Docker Manifest, Schema 2
Media type
    application/vnd.docker.distribution.manifest.v2+json

So we're in a bit of a squeeze here, is there some combination of the docker tooling that's not the latest versions on the Github Actions images? We're using ubuntu-latest.

Expected behaviour

The image should be built without error with files containing non-ACII characters like Æ Ø Å and result in an image artifact with format Docker Manifest, Schema 2.

Actual behaviour

#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.53kB done
#2 DONE 0.0s

#3 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:7.0
#3 DONE 0.2s

#4 [internal] load metadata for docker.io/library/node:14-alpine
#4 ...

#5 [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:7.0
#5 DONE 0.2s

#6 [auth] library/node:pull token for registry-1.docker.io
#6 DONE 0.0s

#4 [internal] load metadata for docker.io/library/node:14-alpine
#4 DONE 1.0s

#7 [internal] load build context
#7 ERROR: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters

#8 [dotnet-build  1/19] FROM mcr.microsoft.com/dotnet/sdk:7.0@sha256:6f1c4809657f4a415401596625c670bb6705346fee7e89c8cbdac567a022efdf
#8 resolve mcr.microsoft.com/dotnet/sdk:7.0@sha256:6f1c4809657f4a415401596625c670bb6705346fee7e89c8cbdac567a022efdf done
#8 sha256:843e4c914a88bb657ab5e1139995b7c4df60687671cd3b04720da5d8b067b3a7 2.01kB / 2.01kB done
#8 sha256:6f1c4809657f4a415401596625c670bb6705346fee7e89c8cbdac567a022efdf 1.79kB / 1.79kB done
#8 CANCELED

#9 [frontend-build 1/7] FROM docker.io/library/node:14-alpine@sha256:434215b487a329c9e867202ff89e704d3a75e554822e07f3e0c0f9e606121b33
#9 resolve docker.io/library/node:14-alpine@sha256:434215b487a329c9e867202ff89e704d3a75e554822e07f3e0c0f9e606121b33 0.0s done
#9 sha256:434215b487a329c9e867202ff89e704d3a75e554822e07f3e0c0f9e606121b33 1.43kB / 1.43kB done
#9 sha256:4e84c956cd276af9ed14a8b2939a734364c2b0042485e90e1b97175e73dfd548 1.16kB / 1.16kB done
#9 CANCELED

#10 [stage-2 1/4] FROM mcr.microsoft.com/dotnet/aspnet:7.0@sha256:c88ac232a58628dacb5d0ccbf0b8fcbf7c97ef24087e449d296616b938f86528
#10 resolve mcr.microsoft.com/dotnet/aspnet:7.0@sha256:c88ac232a58628dacb5d0ccbf0b8fcbf7c97ef24087e449d296616b938f86528 0.0s done
#10 sha256:c88ac232a58628dacb5d0ccbf0b8fcbf7c97ef24087e449d296616b938f86528 1.79kB / 1.79kB done
#10 CANCELED
------
 > [internal] load build context:
------
ERROR: failed to solve: Internal: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters
Error: buildx failed with: ERROR: failed to solve: Internal: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters

Repository URL

No response

Workflow run URL

No response

YAML workflow

name: Build and publish

on:
  push:
    branches:
      - master
    tags:
      - v*

env:
  SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
  DEPLOY_CONFIGURATION: ${{ fromJSON('["app-test.yaml", "app.yaml"]')[startsWith(github.ref, 'refs/tags/v')] }}
  APP_ENGINE_SERVICE: ${{ fromJSON('["app-test", "app-prod"]')[startsWith(github.ref, 'refs/tags/v')] }}

concurrency:
  group: ${{ fromJSON('["test", "prod"]')[startsWith(github.ref, 'refs/tags/v')] }}
  cancel-in-progress: false

jobs:
  build-and-publish:
    name: Build and publish to GCP
    runs-on: ubuntu-latest
    steps:
      - name: Add SHORT_SHA env property with commit short sha
        run: echo "SHORT_SHA=`echo ${github.sha} | cut -c1-8`" >> $GITHUB_ENV

      - uses: actions/checkout@v4

      - name: Authenticate with GCP
        uses: google-github-actions/auth@v2.1.1
        with:
            credentials_json: '${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}'

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v2.1.0
      
      - name: Set up gcr authentication
        run: gcloud auth configure-docker eu.gcr.io
        
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push docker image to gcr
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: eu.gcr.io/application/application:latest
          platforms: linux/amd64
          build-args: |
            GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}
      - name: Deploy image
        run: gcloud app deploy --image-url eu.gcr.io/application/application:latest ${{ env.DEPLOY_CONFIGURATION }}

      - name: Clean up old versions
        run: gcloud app versions list --service=${{ env.APP_ENGINE_SERVICE }} --format="value(version.id)" --sort-by="~version.createTime" | tail -n +6 | xargs -r gcloud app versions delete --quiet

      - name: Send Slack notification
        if: always()
        uses: my-company/slack-notification-action@master
        with:
          status: ${{ job.status }}
          webhook_url: ${{ secrets.MY_COMPANY_BYGGSTATUS_WEBHOOK_URL }}

Workflow logs

No response

BuildKit logs

No response

Additional info

No response

[andmos]

The version info from the workflow:

/usr/bin/docker info
  Client: Docker Engine - Community
   Version:    24.0.7
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.12.1
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.23.3
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 14
   Server Version: 24.0.7
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
   Swarm: inactive
   Runtimes: runc io.containerd.runc.v2
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
   runc version: v1.1.12-0-g51d5e94
   init version: de40ad0
   Security Options:
    apparmor
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.2.0-1019-azure
   Operating System: Ubuntu 22.04.3 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 2
   Total Memory: 6.76GiB
   Name: fv-az576-492
   ID: 625296d6-9be1-4356-a58b-91c3a8df5f0b
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false

[andmos]

I found a workaround here:
docker/buildx#1533
By setting

          provenance: false

in the build-push-action and include setup-buildx-action I'm able to build a Docker Manifest, Schema 2 type image that GCP likes.
I'm leaving this issue open until it's clear that what I'm seeing is expected behavior.

[crazy-max]

@andmos Can you enable debug and post logs please? Would be useful to know what's going on: https://docs.docker.com/build/ci/github-actions/configure-builder/#buildkit-container-logs

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          buildkitd-flags: --debug

Can you also try with latest v0.13 RC of BuildKit if you still repro?

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver-opts: image=moby/buildkit:v0.13.0-rc3
          buildkitd-flags: --debug

Thanks!

[andmos]

@crazy-max sure!
Not sure which of the cases you want to have logs from, since the

ERROR: failed to solve: Internal: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters
Error: buildx failed with: ERROR: failed to solve: Internal: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters

Error only occur when uses: docker/setup-buildx-action isn't present -
If it is present, the build works, but the image results in application/vnd.oci.image.index.v1+json type which Google Cloud App Engine doesn't like - that part is solved by adding provenance: true.

I guess the logs you are after would be from a run without setup-buildx-action right?

Anyways, here is the debug output from a run with

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          buildkitd-flags: --debug

And provenance: false: https://gist.githubusercontent.com/andmos/91a5761d22e9962f9f357e74933f9d09/raw/23696d8e659742508d0c3a3300ecd1ca0e3c2e22/gistfile1.txt

[crazy-max]

Error only occur when uses: docker/setup-buildx-action isn't present

Ok that's because docker/setup-buildx-action creates a container builder to build while if you don't specify it, it will use buildkit inside the docker engine installed on the host and as docker engine doesn't support oci indexes we don't set provenance in buildx, hence why you don't repro without this step in your workflow.

If you still want to use docker/setup-buildx-action you can switch to the docker driver:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver: docker

Or like you did disable provenance but it should work with Cloud Run. This is supported for about a year now: docker/buildx#1533 (comment)

Closing in the meantime but feel free to continue the discussion.

[andmos]

Haha here is your repro I think @crazy-max, with

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver: docker

and

      - name: Build and push docker image to gcr
        uses: docker/build-push-action@v5
        with:
          context: .
          push: ${{ github.ref == 'refs/heads/master' }}
          tags: eu.gcr.io/application/application:latest    
          platforms: linux/amd64
          build-args: |
            GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}

So no provenance, we get

 ERROR: failed to solve: Internal: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters
Error: buildx failed with: ERROR: failed to solve: Internal: rpc error: code = Internal desc = rpc error: code = Internal desc = header key "followpaths" contains value with non-printable ASCII characters

I'll build with logs for you.

Or, I guess I can't add logging since we're using the docker-driver?

[crazy-max]

Or, I guess I can't add logging since we're using the docker-driver?

Yes not possible with docker driver unfortunately.

Can you try by adding a step to set up docker engine so we can have docker engine logs:

      -
        name: Set up Docker
        uses: crazy-max/ghaction-setup-docker@v3
        with:
          version: v24.0.9
          daemon-config: |
            {
              "debug": true
            }
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver: docker

Also check if it works with latest stable docker engine:

      -
        name: Set up Docker
        uses: crazy-max/ghaction-setup-docker@v3
        with:
          version: latest
          daemon-config: |
            {
              "debug": true
            }
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver: docker

Thanks!

[andmos]

For some reason I couldn't get any more logs out of it with "debug": true in the daemon.config file, but I can confirm that the bug is gone in version 25.0.3 of Docker which is the latest stable (installed with your action above).
I guess we'll have to wait for the Docker version of the ubuntu-latest runner-image to update then - it's

Docker Client 24.0.8
Docker Server 24.0.8

now it looks like.

[crazy-max]

bug is gone in version 25.0.3 of Docker which is the latest stable (installed with your action above).

Good to know! fyi @thaJeztah @vvoland @tonistiigi