Slower image builds because of an inconsistent containerd UUID in buildkit

[ollypom]

Describe the bug
Buildkit supports multiple output types. By default in Finch, images get built using buildkit's "docker" output type even though nerdctl's default is the "image" type.

When using the "docker" type, Buildkit will create a tarball (to a docker image format) and then nerdctl will load that tarball into the containerd image store. Removing the tarball steps will provide a significant performance improvements.

$ finch build -t demo .
[+] Building 2.3s (9/10)                                                                                                                                                                
<snip>
 => exporting to docker image format                                                                                                                                               0.7s
 => => exporting layers                                                                                                                                                            0.0s
 => => exporting manifest sha256:c7a206f2dc1ef877aa2d259177383d21f9a6fd5ac9b60b95b4b68f109d81e0d6                                                                                  0.0s
 => => exporting config sha256:91d3769ed7b5b3c3bed9c6d8bb6518ae2a8e9c8ad8a0af70ae98c8935582f477                                                                                    0.0s
 => => sending tarball   

vs

$ finch build -t demo --output type=image,unpack=true .
[+] Building 1.5s (10/10) FINISHED                                                                                                                                                      
<snip>
 => exporting to image                                                                                                                                                             0.0s
 => => exporting layers                                                                                                                                                            0.0s
 => => exporting manifest sha256:c7a206f2dc1ef877aa2d259177383d21f9a6fd5ac9b60b95b4b68f109d81e0d6                                                                                  0.0s
 => => exporting config sha256:91d3769ed7b5b3c3bed9c6d8bb6518ae2a8e9c8ad8a0af70ae98c8935582f477                                                                                    0.0s
 => => naming to docker.io/library/demo:latest                                                                                                                                     0.0s
 => => unpacking to docker.io/library/demo:latest   

The reason why the "docker" type is chosen, is because nerdctl falls back to "docker" if a contained buildkit workers parameters do not match. And by default in Finch, that is the case. The buildkit worker's containerd UUID label does not match containerds UUID..

Steps to reproduce

After getting a shell into the Finch VM:

$ sudo ctr info
{
    "server": {
        "uuid": "613bbaa1-c064-497c-8917-1169147b2c2f",
        "pid": 1225,
        "pidns": 4026531836
    }
}

$ sudo buildctl debug workers --format "{{json .}}" | jq -r '. | .[].labels'
{
  "org.mobyproject.buildkit.worker.containerd.namespace": "finch",
  "org.mobyproject.buildkit.worker.containerd.uuid": "83906608-6cc8-4a5e-b277-35ed70709712",
  "org.mobyproject.buildkit.worker.executor": "containerd",
  "org.mobyproject.buildkit.worker.hostname": "lima-finch",
  "org.mobyproject.buildkit.worker.network": "host",
  "org.mobyproject.buildkit.worker.selinux.enabled": "false",
  "org.mobyproject.buildkit.worker.snapshotter": "overlayfs"
}

After a quick restart of the buildkid daemon, it fixes itself and the UUIDs match:

$ sudo systemctl restart buildkit
$ sudo buildctl debug workers --format "{{json .}}" | jq -r '. | .[].labels'
{
  "org.mobyproject.buildkit.worker.containerd.namespace": "finch",
  "org.mobyproject.buildkit.worker.containerd.uuid": "613bbaa1-c064-497c-8917-1169147b2c2f",
  "org.mobyproject.buildkit.worker.executor": "containerd",
  "org.mobyproject.buildkit.worker.hostname": "lima-finch",
  "org.mobyproject.buildkit.worker.network": "host",
  "org.mobyproject.buildkit.worker.selinux.enabled": "false",
  "org.mobyproject.buildkit.worker.snapshotter": "overlayfs"
}

And we can build to the "image" type by default! :)

$ finch build -t test .
<snip>
 => exporting to image                                                                                                                                                                                                            0.5s
 => => exporting layers                                                                                                                                                                                                           0.4s
 => => exporting manifest sha256:6ac780490dde7e816a55a3c4e4b59117765ad41336583cd95e1bfcfd4c9561b3                                                                                                                                 0.0s
 => => exporting config sha256:fdfc1e6cb0727b08ef324fe6717e42d05649d689eb3070227af41977dac59da0                                                                                                                                   0.0s
 => => naming to docker.io/library/test:latest                                                                                                                                                                                    0.0s
 => => unpacking to docker.io/library/test:latest 

But if we restart the Finch VM (finch vm stop / finch vm start), it goes back to the wrong UUID again and needs a buildkit daemon restart!!! :( I wonder if its a start order thing but I'm unable to find the part of code in Finch / Lima / Buildkit that could control that.

[pendo324]

I think I figured this out. This is actually a pretty interesting issue. I went down the rabbit hole of systemd service ordering, but no matter what I did, nothing worked except restarting buildkit after boot was complete.

Turns out, that's because we restart containerd manually in our provisioning script that makes containerd user the persistent data disk. This restart changes the containerd server UUID. The issue is that we also need to restart buildkit.