Buildkit selects the wrong file from cache

[WolleTD]

So, yesterday I enabled Buildkit by default on a docker builder VM running a recent Arch Linux – one job failed in the night.

That job was building an Ubuntu 16.04 image, adding extra repositories. A similar job for an Ubuntu 18.04 image ran just before that on the same machine. The repository looks something like this:

- \ubuntu18.04
  - Dockerfile
  - repo.list
- \ubuntu16.04
  - Dockerfile
  - repo.list

The Ubuntu 16.04 build now got the repo.list from the Ubuntu 18.04 directory somehow (which contains bionic instead of xenial). Running several prune commands or even deleting /var/lib/docker didn't help and I was able to reproduce this on exactly one other machine, but not on another.

That machine also had the repository already cloned. In the end, it got fixed by touching the repo.list files or deleting and recreating the local repository, but it took me some hours to get to that point.

Edit: I proceeded touching both repo.list files on the third machine with a specified date, so they got the same modification time as well. Now I am able to reproduce the issue here as well. However, the issue only appears when running cd $dir && docker build ., not when running docker build $dir.

Here's a demonstration script: buildkit-borked.tar.gz. This should (and does on my machines) produce a "xenial" output for Ubuntu 18.04 in the first build, but not in the other two.

Both (or all three) tested systems were running this configuration:

$ docker version
Client:
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d28e
 Built:             Wed Aug  4 10:59:01 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d88bc
  Built:            Wed Aug  4 10:58:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.5
  GitCommit:        72cec4be58a9eb6b2910f5d10f1c01ca47d231c0.m
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b638
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
$ docker system info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 20
 Server Version: 20.10.8
 Storage Driver: btrfs
  Build Version: Btrfs v5.13 
  Library Version: 102
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 72cec4be58a9eb6b2910f5d10f1c01ca47d231c0.m
 runc version: v1.0.1-0-g4144b638
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.13.12-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: (varies)
 Total Memory: (varies)
 Name: (varies)
 ID: (varies)
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[tonistiigi]

This has been fixed in #2081