Allow interactively debugging a failed build stage

[ktock]

What is the problem you're trying to solve

Related: moby/buildkit#1053

It seems that there are demands for enabling to interactively debug failed build steps.

Describe the solution you'd like

I believe nerdctl can achieve it because now that it has integrated to BuildKit (rootful/rootless) containerd worker and can directly access to BuildKit cache snapshots.
This allows nerdctl starting containers directly based on the snapshot of a failed step.

nerdctl run -it $SNAPSHOT_KEY_OF_FAILED_BUILD_STEP /bin/sh

Maybe we need to do at least:

• Fix BuildKit to print cache snapshot key of each step.
• Fix nerdctl to enable to start a container based on a snapshot.

What I want to do is automating something like the following:

# mkdir -p /tmp/ctx && cat <<EOF > /tmp/ctx/Dockerfile
FROM ghcr.io/stargz-containers/ubuntu:20.04-org
RUN echo hello > /hello
RUN exit 1
EOF
# nerdctl build -t foo /tmp/ctx
[+] Building 5.7s (6/6) FINISHED                                                                                                                                                                                                                                                         
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                0.1s
 => => transferring dockerfile: 120B                                                                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                   0.1s
 => => transferring context: 2B                                                                                                                                                                                                                                                     0.0s
 => [internal] load metadata for ghcr.io/stargz-containers/ubuntu:20.04-org                                                                                                                                                                                                         2.1s
 => [1/3] FROM ghcr.io/stargz-containers/ubuntu:20.04-org@sha256:adf73ca014822ad8237623d388cedf4d5346aa72c270c5acc01431cc93e18e2d                                                                                                                                                   2.9s
 => => resolve ghcr.io/stargz-containers/ubuntu:20.04-org@sha256:adf73ca014822ad8237623d388cedf4d5346aa72c270c5acc01431cc93e18e2d                                                                                                                                                   0.0s
 => => sha256:5e9250ddb7d0fa6d13302c7c3e6a0aa40390e42424caed1e5289077ee4054709 0B / 187B                                                                                                                                                                                            3.3s
 => => sha256:57671312ef6fdbecf340e5fed0fb0863350cd806c92b1fdd7978adbd02afc5c3 0B / 851B                                                                                                                                                                                            3.3s
 => => sha256:345e3491a907bb7c6f1bdddcf4a94284b8b6ddd77eb7d93f09432b17b20f2bbe 27.26MB / 28.54MB                                                                                                                                                                                    3.3s
 => => extracting sha256:345e3491a907bb7c6f1bdddcf4a94284b8b6ddd77eb7d93f09432b17b20f2bbe                                                                                                                                                                                           0.8s
 => => extracting sha256:57671312ef6fdbecf340e5fed0fb0863350cd806c92b1fdd7978adbd02afc5c3                                                                                                                                                                                           0.0s
 => => extracting sha256:5e9250ddb7d0fa6d13302c7c3e6a0aa40390e42424caed1e5289077ee4054709                                                                                                                                                                                           0.0s
 => [2/3] RUN echo hello > /hello                                                                                                                                                                                                                                                   0.3s
 => ERROR [3/3] RUN exit 1                                                                                                                                                                                                                                                          0.2s
------
 > [3/3] RUN exit 1:
------
Dockerfile:3
--------------------
   1 |     FROM ghcr.io/stargz-containers/ubuntu:20.04-org
   2 |     RUN echo hello > /hello
   3 | >>> RUN exit 1
   4 |     
--------------------
error: failed to solve: process "/bin/sh -c exit 1" did not complete successfully: exit code: 1
# ctr snapshot list
KEY                                                                     PARENT                                                                  KIND      
3qvrvc20pz8qkjq4wtz4utuif                                                                                                                       Active    
950jf24ia5be042by3827ymsc                                                                                                                       Active    
pjpe6jzedsct1sd14yy2vz6ez                                               sha256:3dd8c8d4fd5b59d543c8f75a67cdfaab30aef5a6d99aea3fe74d8cc69d4e7bf2 Committed 
sha256:3dd8c8d4fd5b59d543c8f75a67cdfaab30aef5a6d99aea3fe74d8cc69d4e7bf2 sha256:8d8dceacec7085abcab1f93ac1128765bc6cf0caac334c821e01546bd96eb741 Committed 
sha256:8d8dceacec7085abcab1f93ac1128765bc6cf0caac334c821e01546bd96eb741 sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439 Committed 
sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439                                                                         Committed 
# ctr snapshot view debug pjpe6jzedsct1sd14yy2vz6ez
# ctr snapshot mount /mnt/debug debug
# mount -t overlay overlay /mnt/debug -o index=off,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/4/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs
# ctr run --rm -t --rootfs /mnt/debug debug /bin/bash
# ls /
bin  boot  dev  etc  hello  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
# cat /hello
hello

Additional context

Optionally, we can allow mounting arbitrary images which contains debugging tools (possibly with lazy-pulling).

[ktock]

cc @AkihiroSuda

[AkihiroSuda]

SGTM but we need an interface to get the snapshot id

[AkihiroSuda]

Implemented in #1049