intermediate (build) container image not visible when using buildkit

[jtnord]

With buildkit enabled (the default) if an issue occurs during the build it is no longer trivial to diagnose by firing up a container based on the image of the previous build step.

For example if you have a Dockerfile with many RUN steps and one of them fails it is very useful to identify why they fail by spinning up a container based on the preceeding RUN steps image. before the buildkit change this image id was clearly visible.

I tried various options to --output however no value seemed to have any effect on the output.

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features
• I have uploaded Diagnostics
• Diagnostics ID:

Actual behavior

as shown below there is no mention of the intermediate container IDs

 ☸ docker-desktop:default  master  C:\workarea\source\github\jenkinsci\acceptance-test-harness\src\main\resources\org\jenkinsci\test\acceptance\docker\fixtures\SvnContainer>docker build --no-cache -t svntest .
[+] Building 41.1s (24/24) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 32B                                                                                                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/debian:buster                                                                                                                                                                                         0.0s
 => [internal] load build context                                                                                                                                                                                                                        0.0s
 => => transferring context: 381B                                                                                                                                                                                                                        0.0s
 => CACHED [ 1/20] FROM docker.io/library/debian:buster                                                                                                                                                                                                  0.0s
 => [ 2/20] RUN  export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install -yq apache2 apache2-utils subversion libsvn-dev libapache2-mod-svn subversion-tools locales viewvc openssh-server supervisor && apt-get clean               33.8s
 => [ 3/20] RUN svnadmin create /home/svn                                                                                                                                                                                                                0.4s
 => [ 4/20] ADD ./config/svnserve.conf /home/svn/conf/svnserve.conf                                                                                                                                                                                      0.0s
 => [ 5/20] ADD ./config/passwd /home/svn/conf/passwd                                                                                                                                                                                                    0.0s
 => [ 6/20] RUN addgroup subversion &&     usermod -a -G subversion www-data &&     chown -R www-data:subversion /home/svn &&     chmod -R g+rws /home/svn                                                                                               0.5s
 => [ 7/20] RUN /usr/sbin/a2enmod auth_digest &&     rm /etc/apache2/mods-available/dav_svn.conf                                                                                                                                                         0.5s
 => [ 8/20] ADD ./config/dav_svn.conf /etc/apache2/mods-available/dav_svn.conf                                                                                                                                                                           0.0s
 => [ 9/20] RUN mkdir -p /etc/subversion                                                                                                                                                                                                                 0.5s
 => [10/20] ADD ./config/passwd.htpasswd /etc/subversion/passwd.htpasswd                                                                                                                                                                                 0.0s
 => [11/20] RUN mkdir -p /var/run/sshd                                                                                                                                                                                                                   0.5s
 => [12/20] RUN useradd svnUser -d /home/svnUser &&     mkdir -p /home/svnUser/.ssh &&     chown svnUser /home/svnUser &&     echo "svnUser:test" | chpasswd &&     echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzpxmTW9mH87DMkMSqBrSecoSHVCkKbW5IOO+4u  0.5s
 => [13/20] RUN locale-gen en_US.UTF-8                                                                                                                                                                                                                   0.5s
 => [14/20] RUN mkdir -p /var/log/supervisor                                                                                                                                                                                                             0.5s
 => [15/20] ADD ./config/supervisord.conf /etc/supervisor/conf.d/supervisord.conf                                                                                                                                                                        0.0s
 => [16/20] RUN mkdir -p /svnRepo                                                                                                                                                                                                                        0.5s
 => [17/20] ADD ./svnRepo /svnRepo                                                                                                                                                                                                                       0.0s
 => [18/20] RUN apt-get install -qy curl                                                                                                                                                                                                                 1.8s
 => [19/20] RUN /usr/sbin/apache2 && curl -v http://127.0.0.1/svn                                                                                                                                                                                        0.4s
 => ERROR [20/20] RUN /usr/sbin/apache2 && svn checkout http://127.0.0.1/svn /svnRepo && svn add /svnRepo/* && svn commit -m 'init' /svnRepo/* && echo 'newRev' >> /svnRepo/testOne.txt && svn commit -m 'Rev with changes' /svnRepo/*                   0.5s
------
 > [20/20] RUN /usr/sbin/apache2 && svn checkout http://127.0.0.1/svn /svnRepo && svn add /svnRepo/* && svn commit -m 'init' /svnRepo/* && echo 'newRev' >> /svnRepo/testOne.txt && svn commit -m 'Rev with changes' /svnRepo/*:
docker/for-win#24 0.502 [Wed Jun 02 14:06:51.608485 2021] [so:warn] [pid 7] AH01574: module dav_module is already loaded, skipping
docker/for-win#24 0.504 AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
docker/for-win#24 0.521 svn: E170013: Unable to connect to a repository at URL 'http://127.0.0.1/svn'
docker/for-win#24 0.521 svn: E175002: Unexpected server error 500 'Internal Server Error' on '/svn'
------
executor failed running [/bin/sh -c /usr/sbin/apache2 && svn checkout http://127.0.0.1/svn /svnRepo && svn add /svnRepo/* && svn commit -m 'init' /svnRepo/* && echo 'newRev' >> /svnRepo/testOne.txt && svn commit -m 'Rev with changes' /svnRepo/*]: exit code: 1

compared to

 ☸ docker-desktop:default  master  C:\workarea\source\github\jenkinsci\acceptance-test-harness\src\main\resources\org\jenkinsci\test\acceptance\docker\fixtures\SvnContainer>set DOCKER_BUILDKIT=0

 ☸ docker-desktop:default  master  C:\workarea\source\github\jenkinsci\acceptance-test-harness\src\main\resources\org\jenkinsci\test\acceptance\docker\fixtures\SvnContainer>docker build --no-cache -t svntest .
Sending build context to Docker daemon  18.94kB
Step 1/30 : FROM debian:buster
 ---> 4a7a1f401734
Step 2/30 : RUN  export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install -yq apache2 apache2-utils subversion libsvn-dev libapache2-mod-svn subversion-tools locales viewvc openssh-server supervisor && apt-get clean
 ---> Running in cb67825f6367
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://security.debian.org/debian-security buster/updates/main amd64 Packages [291 kB]
Get:3 http://deb.debian.org/debian buster InRelease [121 kB]

---- snip ----

Removing intermediate container 714400203c47
 ---> 9a1602c77b93
Step 26/30 : RUN /usr/sbin/apache2 && svn checkout http://127.0.0.1/svn /svnRepo && svn add /svnRepo/* && svn commit -m 'init' /svnRepo/* && echo 'newRev' >> /svnRepo/testOne.txt && svn commit -m 'Rev with changes' /svnRepo/*
 ---> Running in 254d1eed1c7b
[Wed Jun 02 14:09:16.463507 2021] [so:warn] [pid 8] AH01574: module dav_module is already loaded, skipping
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
svn: E170013: Unable to connect to a repository at URL 'http://127.0.0.1/svn'
svn: E175002: Unexpected server error 500 'Internal Server Error' on '/svn'
The command '/bin/sh -c /usr/sbin/apache2 && svn checkout http://127.0.0.1/svn /svnRepo && svn add /svnRepo/* && svn commit -m 'init' /svnRepo/* && echo 'newRev' >> /svnRepo/testOne.txt && svn commit -m 'Rev with changes' /svnRepo/*' returned a non-zero code: 1

Expected behavior

as per the non buildkit option I would like to be able to clearly identify an image (or container) for the steps.

Information

• Is it reproducible?

Yes

• Is the problem new?

Yes

• Did the problem appear with an update?

Yes (the introduction of buildkit)

• A reproducible case if this is a bug, Dockerfiles with reproduction inside is best.

FROM debian:buster

RUN useradd user1
RUN useradd user2
RUN useradd user3
RUN useradd user1
RUN useradd user5

this will fail on a duplicate useradd - and this is a very contrived example where the cause of the error will be obvious from the logs and not need you to delve into the container, however consider something like apache where the setup error is caused by an incorrect configuration file much earlier in the setup and is not obvious in the build logs

• Windows Version:

10 21H1

• Docker Desktop Version:

3.3.3

• WSL2 or Hyper-V backend?

WSL2

• Are you running inside a virtualized Windows e.g. on a cloud server or a VM:

Nope

Steps to reproduce the behavior

1. using the docker file from above run docker build -t .
2. when the build fails try and identify the image from the last step that worked so you can fire up a container based on it for some interactive debugging.

[jtnord]

output was not the flag I was looking for, seems like --progress=plain is the magic

[jtnord]

I may be running low on Coffee - but the sha's outputted by --progress=plain are not that of the image so - re-opening.

[tonistiigi]

If you are looking a digest from the output it is image config that is used as an ID in docker.

edit: missed that you were not even referring to the image digest. These are LLB step digests for tracking (we've removed them to avoid confusion). Only build results are exported in BuildKit, when you set the output flag. For debugging, follow moby/buildkit#1472