exporting cache to harbor error

[weiwendi]

buildkit version

buildctl github.com/moby/buildkit v0.9.2
buildkitd github.com/moby/buildkit v0.9.2 a14b4e097ae1dc7514c5febd6d75f742a166ea75

command

buildctl --addr tcp://buildkitd:1234 build --frontend=dockerfile.v0 \
             --local context=. --local dockerfile=. \
             --output type=image,name=hub.aiops.red/common/builder:v4,push=true \
             --export-cache type=registry,mode=max,ref=hub.aiops.red/common/builder:cache \
             --import-cache type=registry,ref=hub.aiops.red/common/builder:cache

errors

 => ERROR exporting cache                                                                                                                                                                            15.8s
 => => preparing build cache for export                                                                                                                                                              13.8s
 => => writing layer sha256:0f7fd9f8d114013b2d0a5b94d7d708fd7f5e76ba47862f755376b3e9bf3a99c3                                                                                                          0.1s
 => => writing layer sha256:116209eb403c9e3d691d48996430b189c9955b2c2784ed94124fcb70973fbbe6                                                                                                          0.0s
 => => writing layer sha256:35a4f197768941ef308d981a94f6d06fb77b9f2ba89dc04d2daf8292ee297315                                                                                                          0.0s
 => => writing layer sha256:38b1453721cb272cf531587daa8f79cc38641a9750dc269d7f2f08feb80f6602                                                                                                          0.0s
 => => writing layer sha256:3d938c97ec6a17ff0d82b6318f14ac4fd32f421eb25945afd3ecbb4df21a8b12                                                                                                          0.1s
 => => writing layer sha256:6ac12068f30ddb70a6c7973db73dfbad98097b209f571d6b33e445f12b53a4c0                                                                                                          1.5s
 => => writing layer sha256:74e27dc593d49a6d728dfe36976cb1469e076fbf3611e501fd030308cd212a80                                                                                                          0.0s
 => => writing layer sha256:780391780e2083631e4e53928ce31e5a28e6e3480b8cdedb02aa4a9f732c7dcf                                                                                                          0.0s
 => => writing layer sha256:90fe46dd819953eb995f9cc9c326130abe9dd0b3993a998e12c01d0218a0b831                                                                                                          0.0s
 => => writing layer sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1                                                                                                          0.0s
 => => writing layer sha256:bbc37f14aded2d49bfac62dfa404755c9f1cadfee2b35933e4906f0054782888                                                                                                          0.0s
 => => writing layer sha256:d00a973d5fda7185a48f71764a6d081156770a44774a9404948ad7b16f193b6a                                                                                                          0.1s
 => => writing layer sha256:e6105bb37144c9a18dd4950f7388e19fdf3855626520789f9a562e83ea91fd18                                                                                                          0.0s
 => => writing config sha256:3dc4daec3225a15d14963ee4fb072083e6af7863729539e72246c9cc63beecad                                                                                                         0.1s
 => => writing manifest sha256:1920fe7cf510f24c9a473dc77db9dbf7673f3a3d2276a53b3a434a6e399f9955                                                                                                       0.1s
 => [auth] common/builder:pull,push token for hub.aiops.red                                                                                                                                    0.0s
------
 > exporting cache:
------
error: failed to solve: error writing manifest blob: failed commit on ref "sha256:1920fe7cf510f24c9a473dc77db9dbf7673f3a3d2276a53b3a434a6e399f9955": unexpected status: 404 Not Found

[dardonkov]

@weiwendi Hello, did you manage to solve this, what was the problem?

[l4d2boomer]

@weiwendi How did you solve the problem?

[crazy-max]

Was also able to repro on demo.goharbor.io: https://github.com/crazy-max/docker-build-push-action/actions/runs/4074732012/jobs/7020226741#step:7:415

#42 writing layer sha256:fb35763663536953b1c62525e2d3d34772e1d1a08b6cf8e2a22185e5f532ccf3 done
#42 writing config sha256:59f681c07b641ee8a546872b52d475be2ae111ded5c060b71e5088f3b231428e
#42 writing config sha256:59f681c07b641ee8a546872b52d475be2ae111ded5c060b71e5088f3b231428e 0.3s done
#42 writing manifest sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65
#42 preparing build cache for export 2.0s done
#42 writing manifest sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65 0.2s done
#42 ERROR: error writing manifest blob: failed commit on ref "sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65": unexpected status: 404 Not Found

Relevant BuildKit logs: https://github.com/crazy-max/docker-build-push-action/actions/runs/4074732012/jobs/7020226741#step:17:708

   time="2023-02-02T13:04:16Z" level=debug msg="do request" request.header.content-type=application/vnd.oci.image.index.v1+json request.header.user-agent=buildkit/v0.11 request.method=PUT spanID=3b4de2307190b3c1 traceID=173eca867cace87aeee153f0c2f2ef43 url="https://demo.goharbor.io/v2/build-push-action/test-docker-action/manifests/cache"
  time="2023-02-02T13:04:17Z" level=debug msg="fetch response received" response.header.connection=keep-alive response.header.content-length=174 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 02 Feb 2023 13:04:17 GMT" response.header.server=nginx response.header.set-cookie="sid=8aa40db2ce950035f960341cd53ab06c; Path=/; HttpOnly" response.header.x-request-id=65e629b4-62bd-4370-b1b9-c325316d2655 response.status="404 Not Found" spanID=3b4de2307190b3c1 traceID=173eca867cace87aeee153f0c2f2ef43 url="https://demo.goharbor.io/v2/build-push-action/test-docker-action/manifests/cache"
  time="2023-02-02T13:04:17Z" level=debug msg="unexpected response" body="{\"errors\":[{\"code\":\"NOT_FOUND\",\"message\":\"artifact build-push-action/test-docker-action@sha256:0269c10e600f3a375f36ddabdbd264ce9503a455f0d0969ce8a00f24eaecc032 not found\"}]}\n" resp="&{404 Not Found 404 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Length:[174] Content-Type:[application/json; charset=utf-8] Date:[Thu, 02 Feb 2023 13:04:17 GMT] Server:[nginx] Set-Cookie:[sid=8aa40db2ce950035f960341cd53ab06c; Path=/; HttpOnly] X-Request-Id:[65e629b4-62bd-4370-b1b9-c325316d2655]] {0xc0025ff300} 174 [] false false map[] 0xc004e17500 0xc0004ff810}" spanID=3b4de2307190b3c1 traceID=173eca867cace87aeee153f0c2f2ef43
  time="2023-02-02T13:04:17Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref \"sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65\": unexpected status: 404 Not Found"
  error writing manifest blob: failed commit on ref "sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65": unexpected status: 404 Not Found
  1 v0.11.2 buildkitd --debug
  github.com/moby/buildkit/cache/remotecache.(*contentCacheExporter).Finalize
  	/src/cache/remotecache/export.go:140
  github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1.1
  	/src/solver/llbsolver/solver.go:605
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
  	/src/solver/llbsolver/solver.go:913
  github.com/moby/buildkit/solver.(*Job).InContext
  	/src/solver/jobs.go:611
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext
  	/src/solver/llbsolver/solver.go:909
  github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1
  	/src/solver/llbsolver/solver.go:586
  golang.org/x/sync/errgroup.(*Group).Go.func1
  	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1594
  
  1 v0.11.2 buildkitd --debug
  main.unaryInterceptor.func1
  	/src/cmd/buildkitd/main.go:576
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler
  	/src/api/services/control/control.pb.go:2440
  google.golang.org/grpc.(*Server).processUnaryRPC
  	/src/vendor/google.golang.org/grpc/server.go:1340
  google.golang.org/grpc.(*Server).handleStream
  	/src/vendor/google.golang.org/grpc/server.go:1713
  google.golang.org/grpc.(*Server).serveStreams.func1.2
  	/src/vendor/google.golang.org/grpc/server.go:965
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1594

[OlivierKessler01]

Yup, seems like Harbor doesn't like it

[gjrtimmer]

I have the same problem

[gjrtimmer]

I have solved it; for everybody who wants to know, you can fix this with the following.

In your --cache-to argument you have to include image-manifest=true.

Buildkit is using its own propertiery format; when you set this option, it will use the standard format which is compatible with all registries. This will cause Harbor to recognize the cache layers and image when Buildkit pushes them.

This is the argument I use, and the cached layers show up in my harbor.

--cache-to=type=registry,ref=$DOCKER_IMAGE_CACHE_TO,mode=max,image-manifest=true

@OlivierKessler01 @crazy-max @l4d2boomer @weiwendi @dardonkov

Issue is solved and can be closed.