403 Forbidden when pushing custom images based on latest node.js 18.x and 20.x distroless images to GCR

[claudiu-muresan-pfa]

We started having issues with latest nodejs 18.x and 20.x distroless images.
I'll add more context and exemplify for nodejs 20.x.

We have a simple Dockerfile defined with following content:

FROM gcr.io/distroless/nodejs20-debian12:latest-amd64@sha256:15f21895ff226ff9a7f4f8c4bdc3430bffd516670c85da9152f6336a22504e37

A GH workflow builds a new image based on above Dockerfile using following actions:

- name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3
      with:
        install: true
    - name: Image tags ${{ inputs.name }}
      run: bash scripts/tags.sh -u https://gcr.io/v2/distroless/${{ inputs.name }}/tags/list -f ${{ inputs.dockerfile }} ${{ inputs.additional_tags_script_args }}
      shell: bash
    - name: Read tags and prepend repository name
      run: |
        REPO="${{ inputs.repo }}"
        TAGS=$(cat .tags | sed "s|^|$REPO:|; s|,|,$REPO:|g")
        echo "tags=$TAGS" >> $GITHUB_ENV
      shell: bash
    - name: Log in to Google Container Registry
      uses: docker/login-action@v3
      with:
        registry: gcr.io
        username: _json_key
        password: ${{ inputs.gcr_json_key }}
    - name: 'Build and push ${{ inputs.name }}'
      uses: docker/build-push-action@v6
      with:
        context: .
        file: ${{ inputs.dockerfile }}
        push: true
        tags: ${{ env.tags }}
        provenance: false

We push our images still in GCR (so not AR) and with the latest two published distroless images we get 403 Forbidden when pushing into GCR:

#4 [1/1] FROM gcr.io/distroless/nodejs20-debian12:latest-amd64@sha256:78e019e3c01ada26c995b3583a9b5e08e1d51b5c4b5bcf72b66d9a21a3f3e924
#4 resolve gcr.io/distroless/nodejs20-debian12:latest-amd64@sha256:78e019e3c01ada26c995b3583a9b5e08e1d51b5c4b5bcf72b66d9a21a3f3e924 done
#4 DONE 0.1s

#5 exporting to image
#5 exporting layers done
#5 exporting manifest sha256:cced5a4ab0bb4ab53a0e908bd08ad31c1dcfe076cc43379520cdaab8aca30306 done
#5 exporting config sha256:88005e1f42192951523e0e45f5b3dad264f9ede248baef64fd687d3a440e1a97 done
#5 pushing layers
#5 ...

#6 [auth] quiet-mechanic-140114/waylay-base/distroless-nodejs-20:pull,push token for gcr.io
#6 DONE 0.0s

#5 exporting to image
#5 ...

#7 [auth] distroless/nodejs20-debian12:pull quiet-mechanic-140114/waylay-base/distroless-nodejs-20:pull,push token for gcr.io
#7 DONE 0.0s

#5 exporting to image
#5 pushing layers 1.3s done
#5 ERROR: failed to push gcr.io/quiet-mechanic-140114/waylay-base/distroless-nodejs-20:0686a2de-20240905: unexpected status from POST request to https://gcr.io/v2/quiet-mechanic-140114/waylay-base/distroless-nodejs-20/blobs/uploads/?mount=sha256:e5035db4cc0ad10587dbf34eb63fff1d8d[460](https://github.com/waylayio/waylay-container-images/actions/runs/10719510282/job/29723752265#step:4:479)b92079b04bc8a1e9d228f306fb6&from=distroless/nodejs20-debian12: 403 Forbidden
------
 > exporting to image:
------
ERROR: failed to solve: failed to push gcr.io/quiet-mechanic-140114/waylay-base/distroless-nodejs-20:0686a2de-20240905: unexpected status from POST request to https://gcr.io/v2/quiet-mechanic-140114/waylay-base/distroless-nodejs-20/blobs/uploads/?mount=sha256:e5035db4cc0ad10587dbf34eb63fff1d8d460b92079b04bc8a1e9d228f306fb6&from=distroless/nodejs20-debian12: 403 Forbidden
Reference
Check build summary support
Error: buildx failed with: ERROR: failed to solve: failed to push gcr.io/quiet-mechanic-140114/waylay-base/distroless-nodejs-20:0686a2de-20240905: unexpected status from POST request to https://gcr.io/v2/quiet-mechanic-140114/waylay-base/distroless-nodejs-20/blobs/uploads/?mount=sha256:e5035db4cc0ad10587dbf34eb63fff1d8d460b92079b04bc8a1e9d228f306fb6&from=distroless/nodejs20-debian12: 403 Forbidden

Last nodejs 20.x distroless image that worked fine is gcr.io/distroless/nodejs20-debian12:latest-amd64@sha256:21db062157b2c2ae2708211b90c7f078f8c84d95847cebfd8216b19e9b5f6c37

[loosebazooka]

This doesn't feel like a distroless issue? Can you push any other new images?

[claudiu-muresan-pfa]

Well, i have rebuild yesterday, just for testing, with gcr.io/distroless/nodejs20-debian12:latest-amd64@sha256:21db062157b2c2ae2708211b90c7f078f8c84d95847cebfd8216b19e9b5f6c37 and that image was pushed without errors.

[loosebazooka]

Yeah I was thinking that those images are already cached on the registry and maybe that's why you may not be hitting these errors? The content of an image seems unrelated to ones ability to push to a registry?

[claudiu-muresan-pfa]

So this is not in any way linked to #1630?

[loosebazooka]

That only applies to the distroless gcp project.

[claudiu-muresan-pfa]

Some updates ...
After testing with a new service account key where we enabled r/w on Artifact Repository, we now get a 404:

#7 [auth] quiet-mechanic-140114/waylay-base/distroless-nodejs-20:pull,push token for gcr.io
#7 DONE 0.0s

#5 exporting to image
#5 pushing layers 1.1s done
#5 pushing manifest for gcr.io/quiet-mechanic-140114/waylay-base/distroless-nodejs-20:2c9cc538-20240906@sha256:cced5a4ab0bb4ab53a0e908bd08ad31c1dcfe076cc43379520cdaab8aca30306
#5 pushing manifest for gcr.io/quiet-mechanic-140114/waylay-base/distroless-nodejs-20:2c9cc538-20240906@sha256:cced5a4ab0bb4ab53a0e908bd08ad31c1dcfe076cc43379520cdaab8aca30306 0.3s done
#5 ERROR: failed to push gcr.io/quiet-mechanic-140114/waylay-base/distroless-nodejs-20:2c9cc538-20240906: failed commit on ref "manifest-sha256:cced5a4ab0bb4ab53a0e908bd08ad31c1dcfe076cc43379520cdaab8aca30306": unexpected status from PUT request to https://gcr.io/v2/quiet-mechanic-140114/waylay-base/distroless-nodejs-20/manifests/2c9cc538-20240906: 404 Not Found
------
 > exporting to image:
------
ERROR: failed to solve: failed to push gcr.io/quiet-mechanic-140114/waylay-base/distroless-nodejs-20:2c9cc538-20240906: failed commit on ref "manifest-sha256:cced5a4ab0bb4ab53a0e908bd08ad31c1dcfe076cc43379520cdaab8aca30306": unexpected status from PUT request to https://gcr.io/v2/quiet-mechanic-140114/waylay-base/distroless-nodejs-20/manifests/2c9cc538-20240906: 404 Not Found
Reference
Check build summary support
Error: buildx failed with: ERROR: failed to solve: failed to push gcr.io/quiet-mechanic-140114/waylay-base/distroless-nodejs-20:2c9cc538-20240906: failed commit on ref "manifest-sha256:cced5a4ab0bb4ab53a0e908bd08ad31c1dcfe076cc43379520cdaab8aca30306": unexpected status from PUT request to https://gcr.io/v2/quiet-mechanic-140114/waylay-base/distroless-nodejs-20/manifests/2c9cc538-20240906: 404 Not Found

As mentioned before, we are still using GCR and plan to migrate to AR in the next months.

Our dockerfile being so simple as:

FROM gcr.io/distroless/nodejs20-debian12:latest-amd64@sha256:15f21895ff226ff9a7f4f8c4bdc3430bffd516670c85da9152f6336a22504e37

, probably all layers get cached and the docker build action sees that the google distroless image is on artifact repository and also that we try to push to AR and tries to optimise by referencing layers

We chose to use a workaround - changed the dockerfile by adding a new dummy layer (also reverted back to the service account key that only allows pushing images to GCR):

FROM gcr.io/distroless/nodejs20-debian12:latest-amd64@sha256:78e019e3c01ada26c995b3583a9b5e08e1d51b5c4b5bcf72b66d9a21a3f3e924

# Workaround until migration to AR is completed
# Adds a dummy layer to ensure Docker does not use build cache
LABEL build_no_cache="true"
COPY no-cache.txt /no-cache.txt

We see this as a workaround and need to re-test once the migration to AR is completed on our side.

Do you see any other solutions?

[loosebazooka]

Maybe this is a bug with AR. Lemme pull someone in here: @ardagnir is there any chance of gcr/ar cross caching that could arise from this? These are not the same gcp project. (and correct me if I'm wrong @claudiu-muresan-pfa) it's basically doing a image copy operation through another machine?

[loosebazooka]

does only using the label work? Or do you need to add a layer?

[claudiu-muresan-pfa]

Need to add a layer. It’s not working just with the label.

[loosebazooka]

Just an update here, someone is looking into this on the service side

[mattmoor]

If I had to guess, the problem is very likely that GCR cannot mount a repo backed by AR.