-
Notifications
You must be signed in to change notification settings - Fork 491
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GitHub Actions cache fails with cache key: blob not found #681
Comments
It seems to me like the error is coming from buildkit, at least that is the only place where I can find this snippet: ce, err := p.ci.cache.Load(ctx, key)
if err != nil {
return nil, err
}
if ce == nil {
return nil, errors.Errorf("blob not found")
} Should I open an issue in that repository? |
@crazy-max If you're looking into this, the question is why https://github.com/moby/buildkit/pull/1498/files doesn't work properly in this case and fall back gracefully(assuming blob is legit and there is no other corruption). I guess it might be because |
Agree, I also think it might be a rough case where @jojomatik is using the @jojomatik Can you enable debug in your workflow for the - name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
with:
version: v0.6.0
buildkitd-flags: --debug |
Enable debug output to provide more information for cache restore issue (docker/buildx#681).
Yes, here you go: jojomatik/blockcluster@2a26e2a (workflow). Edit: I don't see any difference. Not sure if it worked?
I believe I checked the docs at least to see if I'm exceeding some sort of storage quota and think I was well below that, but it's entirely possible that either
But I think in either way it shouldn't just fail but rather rebuild that stage, don't you think? On another note: the last steps (14 - 16) of the
Not sure if these might be connected but I remember that I had one failed workflow in the past. It failed as I had already merged the PR and deleted the branch before the workflows were complete. Maybe that one corrupted the cache somehow? |
See https://github.com/docker/setup-buildx-action#buildkit-container-logs
Yes errors should be ignored by cache loader. That's an issue that needs to be fixed on BuildKit like @tonistiigi said.
Can you use a fresh cache-from: type=gha,scope=blockcluster
cache-to: type=gha,scope=blockcluster |
Use blockcluster scope for caching docker layers to check if it helps with the caching issue docker/buildx#681.
Thanks :)
Yes: jojomatik/blockcluster@f180097 (workflow). That one worked fine, as no cache is available. Should I try to rerun it to see whether the cache is loaded on a second run? In case it helps: I just had a look, the other caches for frontend and backend CI seem to be around 160MB (2 * 80MB as I cache |
Yes please. |
Rerun complete: https://github.com/jojomatik/blockcluster/runs/3124430755 Not sure if it's representative though, as it didn't have to decompress (not sure if this is the right term, I hope you know what I mean) the cache, to build further layers. It just realized that every layer was already cached and pushed it to the registry again. I've gone ahead and added a small change (jojomatik/blockcluster@511a3f1) that should trigger a new workflow with a new vue rebuild (builder 14/16) as the commit sha changed. Seems like this worked as well. Does this confirm that my cache was somehow corrupted? |
I'm also seeing this issue on a multi-stage image build. As you can see below it doesn't necessarily happen when doing cross-stage copying:
Also probably worth mentioning is that these jobs run on self-hosted runners. |
We are seeing this as well on shared runners.
The corresponding action log is here https://github.com/caos/zitadel/runs/3175870670?check_suite_focus=true#step:4:195 We see this across different branches and jobs. |
Thanks for your feedback everyone.
@jojomatik Blobs are somehow missing but it should not fail in this case actually. We are going to investigate on this issue. |
There are still some bugs in the `gha` cache backend that are causing builds to fail: - docker/buildx#681 - docker/build-push-action#422 Removing caching until upstream resolves these issues.
Run into probably the same issue – with error: failed to solve: failed to compute cache key: blob not found
Error: buildx failed with: error: failed to solve: failed to compute cache key: blob not found It fails on FROM node:14-buster-slim as packages
WORKDIR /build
COPY . . And we use: context: .
file: api/Dockerfile |
Same issue here, out of nowhere the builds start failing. Disabling the cache is our current workaround. |
There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling limits. Background reading: - buildx failed with: error: failed to solve: blob not found #422 docker/build-push-action#422 - Copy from previous stage fails #681 docker/buildx#681 - GHA export cache fails with mode=max #2276 moby/buildkit#2276 Try to work around this by limiting the overall concurrency to 1 for this workflow.
There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling limits. Background reading: - buildx failed with: error: failed to solve: blob not found #422 docker/build-push-action#422 - Copy from previous stage fails #681 docker/buildx#681 - GHA export cache fails with mode=max #2276 moby/buildkit#2276 Try to work around this by limiting the overall concurrency to 1 for this workflow.
There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling limits. Background reading: - buildx failed with: error: failed to solve: blob not found #422 docker/build-push-action#422 - Copy from previous stage fails #681 docker/buildx#681 - GHA export cache fails with mode=max #2276 moby/buildkit#2276 Try to work around this by limiting the overall concurrency to 1 for this workflow.
There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling limits. Background reading: - buildx failed with: error: failed to solve: blob not found #422 docker/build-push-action#422 - Copy from previous stage fails #681 docker/buildx#681 - GHA export cache fails with mode=max #2276 moby/buildkit#2276 Try to work around this by limiting the overall concurrency to 1 for this workflow.
Same issue here (private project). We have been using the We have reverted to using manual buildx cache and it works but isn’t really much of an improvement compared to not caching at all.
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: docker/setup-buildx-action@v1
- uses: docker/login-action@v1
with:
registry: ghcr.io
- uses: docker/build-push-action@v2
with:
context: .
push: true
tags: ghcr.io/…
cache-from: type=gha
cache-to: type=gha,mode=max |
There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling limits. Background reading: - buildx failed with: error: failed to solve: blob not found #422 docker/build-push-action#422 - Copy from previous stage fails #681 docker/buildx#681 - GHA export cache fails with mode=max #2276 moby/buildkit#2276 Try to work around this by limiting the overall concurrency to 1 for this workflow.
Caching is currently causing the build to fail See docker/buildx#681
Should be fixed since BuildKit 0.9.1. |
Thanks for the info and the good work! 🎉 |
If you encounter this kind of issue with GHA cache again please open an issue on BuildKit repo as this is mostly a backend issue. Thanks. |
Do we need to specify this in the with:
driver-opts: image=moby/buildkit:0.9.1 |
Not needed the default tag has been updated. |
Ran into this today with v0.13.0 buildkit |
@KevinMind and others reacting: This is a closed issue, please don't use it to report a regression in a new release. If it is same as moby/buildkit#4765 you can post your full issue details in there. There is a candidate fix in moby/buildkit#4771 for testing. |
Copying from one stage to another failed in one of my repositories today in a Github Actions workflow after a minor change (jojomatik/blockcluster@cdeb83a) in a workflow file, which shouldn't affect docker at all: https://github.com/jojomatik/blockcluster/runs/3116972956
I'm not sure where exactly this error message comes from and I wasn't able to pin it down with a google search either. So I'm not entirely sure if this is a buildx issue or an issue with one of its dependencies?
This is the relevant section in the workflow file: https://github.com/jojomatik/blockcluster/blob/cdeb83a457f28692cc162e2453a8fe4a36be4ec8/.github/workflows/publish.yml#L92-L111
I've since tried to update it to use the non-rc versions (jojomatik/blockcluster@d4488ba), but it has the same issues. Yesterday those workflows worked like a charm.
If I open up this issue in the wrong repository, I'm sorry. I'm not yet fully comprehending how all these docker parts work together :)
I'd be grateful if you'd find some time to look into this, Thank you! If you need additional information, feel free to ask. I couldn't find a PR template though.
The text was updated successfully, but these errors were encountered: