Occasional "unexpected EOF" errors during multi stage docker build

[vgrigoruk]

We see occasional unexpected EOF errors when running multi-stage docker builds with buildkit enabled (unfortunately not 100% reproducible). I guess it was supposed to be fixed by moby/moby#40993, but it looks like there are some edge cases that are not yet fixed. Crash logs are attached

DOCKER_BUILDKIT=1 docker build --pull --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "REDACTED:master" -t "REDACTED:latest"

#1 [internal] load build definition from Dockerfile
  | #1 transferring dockerfile: 38B done
  | #1 DONE 0.0s
  |  
  | #2 [internal] load .dockerignore
  | #2 transferring context: 35B done
  | #2 DONE 0.0s
  |  
  | #4 [internal] load metadata for public.ecr.aws/peakon/node:14.15.4-buster-b...
  | unexpected EOF

Dockerfile

FROM public.ecr.aws/peakon/node:14.15.4-buster-builder AS builder

FROM public.ecr.aws/peakon/node:14.15.4-buster-runtime AS web

COPY --from=builder --chown=peakon /app/node_modules node_modules

COPY --chown=peakon . .

RUN rm -f .npmrc

CMD [ "node", "start_web.js" ]

FROM web as clock

CMD [ "node", "start_clock.js" ]

FROM web as worker

CMD [ "node", "start_worker.js" ]

FROM web as release

CMD [ "npm", "run", "migrate:fast" ]

FROM web

docker system info

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 205
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.214-160.339.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.805GiB
 Name: ip-10-4-0-129.eu-west-1.compute.internal
 ID: JPEQ:M2FL:7ELD:4FF4:UHIW:PU2E:OTJX:CEOW:OD3Q:PSOA:QQH4:XJC7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

dockerd log

dockerd-crash.log

[tonistiigi]

Fixed in containerd/containerd#4855, need to update vendor

[vgrigoruk]

Looks like it depends on #1921

[tonistiigi]

This has been merged in master.