buildkitd fails to resolve digest for third party registries

[seemethere]

Summary

buildkitd fails to build when doing a FROM ${image from third party registry}

Versions

Versions for this were built directly from current master (a61393a as of time of reporting)

buildctl:

❯ buildctl --version
buildctl github.com/moby/buildkit v0.5.1-72-ga61393a5 a61393a5bc78d34458bc1d1b60dac16cfdec92db

buildkitd:

/ # buildkitd --version
buildkitd github.com/moby/buildkit v0.5.1-72-ga61393a5 a61393a5bc78d34458bc1d1b60dac16cfdec92db

Reproduction case

1. Do sudo make && make images && sudo make install first

2. Then create this Dockerfile:

Dockerfile:

FROM registry.access.redhat.com/ubi8/ubi:8.0
RUN echo "hello"

3. Then run to start the buildkitd:

docker run -d --name buildkitd --privileged -p 1234:1234 moby/buildkit:local --addr tcp://0.0.0.0:1234

4. The run this to build the Dockerfile:

❯ BUILDKIT_HOST=tcp://0.0.0.0:1234 buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=.
[+] Building 2.3s (4/5)
 => [internal] load .dockerignore                                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                                0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                           0.0s
 => => transferring dockerfile: 158B                                                                                                                                                                           0.0s
 => ERROR [internal] load metadata for registry.access.redhat.com/ubi8/ubi:8.0                                                                                                                                 1.8s
 => ERROR [1/2] FROM registry.access.redhat.com/ubi8/ubi:8.0                                                                                                                                                   0.5s
 => => resolve registry.access.redhat.com/ubi8/ubi:8.0                                                                                                                                                         0.5s
------
 > [internal] load metadata for registry.access.redhat.com/ubi8/ubi:8.0:
------
------
 > [1/2] FROM registry.access.redhat.com/ubi8/ubi:8.0:
------
error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: could not resolve digest for registry.access.redhat.com/ubi8/ubi:8.0

`buildkitd` logs:

❯ cat logs
time="2019-06-27T21:01:57Z" level=info msg="found worker \"uolg1agqt31684h9na8y3gvfs\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:f596b2e7f26d org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64]"
time="2019-06-27T21:01:57Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2019-06-27T21:01:57Z" level=info msg="found 1 workers, default=\"uolg1agqt31684h9na8y3gvfs\""
time="2019-06-27T21:01:57Z" level=warning msg="currently, only the default worker can be used."
time="2019-06-27T21:01:57Z" level=info msg="running server on [::]:1234"
time="2019-06-27T21:02:15Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: could not resolve digest for registry.access.redhat.com/ubi8/ubi:8.0
github.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve
        /src/vendor/github.com/containerd/containerd/remotes/docker/resolver.go:237
github.com/moby/buildkit/util/pull.(*cachedResolver).Resolve
        /src/util/pull/resolver.go:145
github.com/moby/buildkit/util/pull.(*Puller).Resolve.func1
        /src/util/pull/pull.go:75
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:44
github.com/moby/buildkit/util/pull.(*Puller).Resolve
        /src/util/pull/pull.go:51
github.com/moby/buildkit/source/containerimage.(*puller).CacheKey
        /src/source/containerimage/pull.go:144
github.com/moby/buildkit/solver/llbsolver/ops.(*sourceOp).CacheMap
        /src/solver/llbsolver/ops/source.go:61
github.com/moby/buildkit/solver.(*sharedOp).CacheMap.func1
        /src/solver/jobs.go:635
github.com/moby/buildkit/util/flightcontrol.(*call).run
        /src/util/flightcontrol/flightcontrol.go:116
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:44
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
failed to load cache key
github.com/moby/buildkit/solver.(*edge).unpark.func1
        /src/solver/edge.go:335
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
        /src/solver/internal/pipe/pipe.go:78
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
failed to build LLB
github.com/moby/buildkit/solver/llbsolver.(*llbBridge).Solve
        /src/solver/llbsolver/bridge.go:101
github.com/moby/buildkit/frontend/gateway/forwarder.(*bridgeClient).Solve
        /src/frontend/gateway/forwarder/forward.go:44
github.com/moby/buildkit/frontend/dockerfile/builder.Build.func3.1
        /src/frontend/dockerfile/builder/build.go:379
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:58
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
failed to solve with frontend dockerfile.v0
github.com/moby/buildkit/solver/llbsolver.(*llbBridge).Solve
        /src/solver/llbsolver/bridge.go:112
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve
        /src/solver/llbsolver/solver.go:133
github.com/moby/buildkit/control.(*Controller).Solve
        /src/control/control.go:276
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1
        /src/api/services/control/control.pb.go:1364
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1
        /src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:510
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
        /src/api/services/control/control.pb.go:1366
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:972
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1252
google.golang.org/grpc.(*Server).serveStreams.func1.1
        /src/vendor/google.golang.org/grpc/server.go:691
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337"

[seemethere]

Looks like this is related to containerd/containerd#3238,

Should be resolved with containerd/containerd#3245

[seemethere]

It appears as though a revendor to containerd/containerd@7c1e883 should resolve the issue