Rootful BuildKit + SELinux: Dockerfile RUN command throws error code 139

[AkihiroSuda]

When SELinux is enforcing and BuildKit is running as Rootful mode, every Dockerfile RUN command throws error code 139.
Rootless mode seems unaffected.

# getenforce 
Enforcing

# cat Dockerfile 
FROM alpine
RUN ls

# buildctl build --frontend dockerfile.v0 --local dockerfile=. --local context=.
[+] Building 2.9s (5/5) FINISHED                                                                                                                                                                                                                                         
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                0.1s
 => => transferring dockerfile: 56B                                                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                                                    2.7s
 => CACHED [1/2] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                                                                                                                                              0.0s
 => => resolve docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                                                                                                                                                     0.0s
 => ERROR [2/2] RUN ls                                                                                                                                                                                                                                              0.1s
------
 > [2/2] RUN ls:
------
Dockerfile:2
--------------------
   1 |     FROM alpine
   2 | >>> RUN ls
   3 |     
--------------------
error: failed to solve: process "/bin/sh -c ls" did not complete successfully: exit code: 139

Daemon logs:

# buildkitd
INFO[2022-10-18T14:05:52Z] auto snapshotter: using overlayfs            
WARN[2022-10-18T14:05:52Z] using host network as the default            
WARN[2022-10-18T14:05:52Z] git source cannot be enabled: failed to find git binary: exec: "git": executable file not found in $PATH 
INFO[2022-10-18T14:05:52Z] found worker "3to1fo25rmz1hwacdyyyu5kog", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:lima-fedora org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/386] 
WARN[2022-10-18T14:05:52Z] using host network as the default            
WARN[2022-10-18T14:05:52Z] git source cannot be enabled: failed to find git binary: exec: "git": executable file not found in $PATH 
INFO[2022-10-18T14:05:52Z] found worker "k8jntgsnmdl8m2e15pqv60sem", labels=map[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproject.buildkit.worker.containerd.uuid:4aa6134d-7969-409c-a3b4-8216a502d73b org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:lima-fedora org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/386] 
INFO[2022-10-18T14:05:52Z] found 2 workers, default="3to1fo25rmz1hwacdyyyu5kog" 
WARN[2022-10-18T14:05:52Z] currently, only the default worker can be used. 
INFO[2022-10-18T14:05:52Z] running server on /run/buildkit/buildkitd.sock 
INFO[2022-10-18T14:06:06Z] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf 
ERRO[2022-10-18T14:06:07Z] /moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = process "/bin/sh -c ls" did not complete successfully: exit code: 139

Version info:

# uname -a
Linux lima-fedora 5.19.15-201.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Oct 13 18:58:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa | grep -i selinux | sort
container-selinux-2.190.0-1.fc36.noarch
libselinux-3.3-4.fc36.x86_64
libselinux-utils-3.3-4.fc36.x86_64
python3-libselinux-3.3-4.fc36.x86_64
rpm-plugin-selinux-4.17.1-3.fc36.x86_64
selinux-policy-36.16-1.fc36.noarch
selinux-policy-targeted-36.16-1.fc36.noarch

# buildkitd --version
buildkitd github.com/moby/buildkit v0.10.4 a2ba6869363812a210fcc3ded6926757ab780b5f

# runc --version
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d1
spec: 1.0.2-dev
go: go1.19.1
libseccomp: 2.5.1

Originally reported by @pcramasamy in

• [Rootful + SELinux] [BuildKit] RUN command throws error code 139 on centos containerd/nerdctl#1413

[rhatdan]

Need AVC messages
sudo ausearch -m avc

[AkihiroSuda]

Need AVC messages sudo ausearch -m avc

time->Wed Oct 19 09:32:07 2022
type=AVC msg=audit(1666171927.377:524): avc:  denied  { map } for  pid=1685 comm="sh" path="/bin/busybox" dev="overlay" ino=66362 scontext=system_u:system_r:container_t:s0:c105,c928 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

[rhatdan]

Where is /bin/busybox stored? Looks like it is under /var/lib?

[AkihiroSuda]

Where is /bin/busybox stored? Looks like it is under /var/lib?

"/var/lib/buildkit/runc-overlayfs/executor/3207p4zbtapbr152e79hxo85v/rootfs", which is mounted from owerdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/3/fs,upperdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/5/fs,workdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/5/work (when running with overlayfs snapshotter).

The snapshotter implementation is basically same as containerd but with the different directories.

[rhatdan]

Ok so it is a container engine.