GHA/Local cache doesn't work

[dmatviichuk]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

Cache reuse doesn't work on self-hosted runner. Cache is being created but docker doesn't use it. I tried two options: gha and local cache with manual exporting to GH Cache.

Expected behaviour

Docker uses cache

Actual behaviour

GHA/Local cache doesn't work

Repository URL

No response

Workflow run URL

No response

YAML workflow

- name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Set up Docker cache
        uses: actions/cache@v4
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./docker/Dockerfile.github
          push: true
          tags: ${{ env.ECR_REGISTRY }}/${{ env.ECR_REPOSITORY }}:${{ needs.setup.outputs.release }}
          build-args: |
            PIP_GH_TOKEN=${{ secrets.ADVINOW_GITHUB_TOKEN }}
            REQUIREMENTS_FILE=requirements.${{ needs.setup.outputs.environment }}-github.txt
            ENVIRONMENT=${{ needs.setup.outputs.environment }}
            DD_GIT_REPOSITORY_URL=${{ github.server_url }}/${{ github.repository }}
            DD_GIT_COMMIT_SHA=${{ github.sha }}
          cache-from: type=gha,scope=${{ needs.setup.outputs.environment }}
          cache-to: type=gha,mode=max,scope=${{ needs.setup.outputs.environment }}

Workflow logs

Run docker/build-push-action@v5
GitHub Actions runtime token ACs
Docker info
Proxy configuration
Buildx version
/usr/bin/docker buildx build --build-arg PIP_GH_TOKEN=*** --build-arg REQUIREMENTS_FILE=requirements.development-github.txt --build-arg ENVIRONMENT=development --build-arg DD_GIT_REPOSITORY_URL=https://github.com/advinow/ext-data-broker --build-arg DD_GIT_COMMIT_SHA=7ceb4fdeddd53167164d82f6ec22bf0757b208c3 --cache-from type=gha,scope=848070539938.dkr.ecr.us-west-2.amazonaws.com/ext-data-broker:edb-2852 --cache-to type=gha,mode=max,scope=848070539938.dkr.ecr.us-west-2.amazonaws.com/ext-data-broker:edb-2852 --file ./docker/Dockerfile.github --iidfile /actions-runner/_work/_temp/docker-actions-toolkit-JOlHjf/iidfile --provenance mode=min,inline-only=true,builder-id=https://github.com/advinow/ext-data-broker/actions/runs/808881[33](https://github.com/advinow/ext-data-broker/actions/runs/8088813326/job/22103619775#step:11:34)26 --tag 848070539938.dkr.ecr.us-west-2.amazonaws.com/ext-data-broker:edb-2852 --metadata-file /actions-runner/_work/_temp/docker-actions-toolkit-JOlHjf/metadata-file --push .
#0 building with "builder-3c6419cb-3f1f-4061-afb9-f76a6c8b1e5d" instance using docker-container driver

#1 [internal] load build definition from Dockerfile.github
#1 transferring dockerfile: 1.44kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/python:3.10-slim
#2 DONE 1.1s

#3 [internal] load .dockerignore
#3 transferring context: 120B done
#3 DONE 0.0s

#4 [internal] load build context
#4 DONE 0.0s

#5 [ 1/13] FROM docker.io/library/python:3.10-slim@sha256:4bd9a0e5da619b155a214342bb4fdc774a947fec9061016a123fa4073431c9cb
#5 resolve docker.io/library/python:3.10-slim@sha256:4bd9a0e5da619b155a214342bb4fdc774a947fec9061016a123fa4073431c9cb 0.0s done
#5 DONE 0.0s

#6 importing cache manifest from gha:15103199574277909574
#6 DONE 0.4s

#4 [internal] load build context
#4 transferring context: 27.35MB 2.2s done
#4 DONE 2.2s

#5 [ 1/13] FROM docker.io/library/python:3.10-slim@sha256:4bd9a0e5da619b155a214342bb4fdc774a947fec9061016a123fa4073431c9cb
#5 sha256:d0d45da63dd11d6510595eb29c9614e5f3e669e38bfbf94a[36](https://github.com/advinow/ext-data-broker/actions/runs/8088813326/job/22103619775#step:11:38)2ff156d81739ca 3.36MB / 3.36MB 0.3s done
#5 sha256:ce524da9d5723ff0cc7d540777cd1202c07fad525cfada0c16924d447d482d5f 242B / 242B 0.3s done
#5 sha256:c0d8da8ab021fce179c54d4f899b83f7fdb5d5bd7ec73b737bcb3f60c00908f9 12.38MB / 12.38MB 0.5s done
#5 sha256:51d1f07906b71fd60ac43c61035514996a8ad8dbfd39d4f570ac5446b064ee5d 3.51MB / 3.51MB 0.3s done
#5 sha256:e1caac4eb9d2ec24aa3618e5992208321a92492aef5fef5eb9e470895f771c56 29.12MB / 29.12MB 0.7s done
#5 extracting sha256:e1caac4eb9d2ec24aa3618e5992208321a92492aef5fef5eb9e470895f771c56
#5 extracting sha256:e1caac4eb9d2ec24aa3618e5992208321a92492aef5fef5eb9e470895f771c56 3.3s done
#5 DONE 4.6s

#5 [ 1/13] FROM docker.io/library/python:3.10-slim@sha256:4bd9a0e5da619b155a2[143](https://github.com/advinow/ext-data-broker/actions/runs/8088813326/job/22103619775#step:11:148)42bb4fdc774a947fec9061016a123fa4073431c9cb
#5 extracting sha256:51d1f07906b71fd60ac43c61035514996a8ad8dbfd39d4f570ac5446b064ee5d
#5 extracting sha256:51d1f07906b71fd60ac43c61035514996a8ad8dbfd39d4f570ac5446b064ee5d 0.4s done
#5 DONE 5.0s

#5 [ 1/13] FROM docker.io/library/python:3.10-slim@sha256:4bd9a0e5da619b155a214342bb4fdc774a947fec9061016a123fa4073431c9cb
#5 extracting sha256:c0d8da8ab021fce179c54d4f899b83f7fdb5d5bd7ec73b737bcb3f60c00908f9
#5 extracting sha256:c0d8da8ab021fce179c54d4f899b83f7fdb5d5bd7ec73b737bcb3f60c00908f9 1.3s done
#5 DONE 6.3s

#5 [ 1/13] FROM docker.io/library/python:3.10-slim@sha256:4bd9a0e5da619b155a214342bb4fdc774a947fec9061016a123fa4073431c9cb
#5 extracting sha256:ce524da9d5723ff0cc7d540777cd1202c07fad525cfada0c16924d447d482d5f 0.0s done
#5 extracting sha256:d0d45da63dd11d6510595eb29c9614e5f3e669e38bfbf94a362ff156d81739ca
#5 extracting sha256:d0d45da63dd11d6510595eb29c9614e5f3e669e38bfbf94a362ff156d81739ca 0.7s done
#5 DONE 6.9s

#7 [ 2/13] RUN     apt-get update -y &&     apt-get install -y libpq-dev gcc build-essential git poppler-utils logrotate jq &&     apt-get autoclean -y
#7 0.204 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
#7 0.240 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
#7 0.258 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#7 0.458 Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8786 kB]
#7 0.671 Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [12.7 kB]
#7 0.835 Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [[144](https://github.com/advinow/ext-data-broker/actions/runs/8088813326/job/22103619775#step:11:149) kB]

BuildKit logs

No response

Additional info

No response

[crazy-max]

Do you have a link to a repository with minimal repro? That would help to understand. Also post the debug logs. Thanks.

[dmatviichuk]

11_Build and push.txt
debug logs were attached. We can see that cache was pulled from GHA importing cache manifest from gha:15082751284183973667 but next stages didn't use it.

Rerun same pipeline, cache works:
#5 [ 1/13] FROM docker.io/library/python:3.10-slim@sha256:4bd9a0e5da619b155a214342bb4fdc774a947fec9061016a123fa4073431c9cb
#5 resolve docker.io/library/python:3.10-slim@sha256:4bd9a0e5da619b155a214342bb4fdc774a947fec9061016a123fa4073431c9cb 0.0s done
#5 DONE 0.0s

#6 importing cache manifest from gha:13113280659452058020
#6 DONE 0.8s

#4 [internal] load build context
#4 transferring context: 27.35MB 1.3s done
#4 DONE 1.3s

#7 [ 4/13] COPY pip.conf /root/.pip/pip.conf
#7 CACHED

#8 [ 7/13] RUN pip install --no-cache-dir --upgrade pip --src /usr/local/src && pip install --force-reinstall --no-cache-dir -r /tmp/requirements/requirements.development-github.txt --src /usr/local/src
#8 CACHED

#9 [11/13] RUN chmod +x /app/start_charting_process_ddtrace.sh
#9 CACHED

#10 [12/13] COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
#10 CACHED

#11 [ 2/13] RUN apt-get update -y && apt-get install -y libpq-dev gcc build-essential git poppler-utils logrotate jq && apt-get autoclean -y
#11 CACHED

[crazy-max]

Also post the debug logs. Thanks.