docker:19.03.6-dind: chown not working with "docker build" when dockerd is run in userns

[jappenzesr]

When configuring dockerd in 19.03.6-dind to run in a user-namespace, chown doesn't work anymore.

Why is it relevant
We use dind to provide docker building services in a CI/CD platform shared among thousands of developers. Running docker (even in dind) in a user-namespace is an essential part of the security strategy to reduce the attack vector on the underlying build node.

Testcase

1. Create an image with dockerd run in userns, based on 19.03.6-dind

FROM docker.artifact.swissre.com/docker:19.03.6-dind
RUN mkdir /etc/docker && echo '{ "userns-remap": "dockerns:dockerns" }' > /etc/docker/daemon.json && \
    addgroup -g 1000 dockerns && adduser dockerns -u 1000 -D -H -G dockerns -s /sbin/nologin && \
    echo "dockerns:1000:65536" > /etc/subuid && echo "dockerns:1000:65536" > /etc/subgid
ENTRYPOINT ["dockerd-entrypoint.sh"]

2. Build the image (tag 'my-dind')
3. Run it

docker run --privileged --userns=host --name some-docker -d \
           --network some-network --network-alias docker \
           -e DOCKER_TLS_CERTDIR=/certs \
           -v some-docker-certs-ca:/certs/ca \
           -v some-docker-certs-client:/certs/client \
           my-dind

4. Run the client in a second container, and connect to it via interactive shell

docker run -it --rm --network some-network \
    -e DOCKER_TLS_CERTDIR=/certs \
    -v some-docker-certs-client:/certs/client:ro \
    docker:latest sh

5. In the client, create the below Dockerfile

FROM docker-hub.artifact.swissre.com/ubuntu:latest
RUN useradd -m testuser
RUN chown testuser /home/testuser
RUN ls -la /home/testuser
USER testuser
RUN touch /home/testuser/testfile
RUN ls -la /home/testuser/testfile

Expected outcome
The docker build should complete successfully. The users home-directory should be created and owned by testuser, and it should be possible to create a file in that directory after USER testuser.
This is indeed the case with a pure 19.03.6-dind setup

# Create Dockerfile in /tmp as per above
/tmp # docker build .
Sending build context to Docker daemon  2.048kB
Step 1/7 : FROM docker-hub.artifact.swissre.com/ubuntu:latest
 ---> 72300a873c2c
Step 2/7 : RUN useradd -m testuser
 ---> Running in 713f1bf1e37e
Removing intermediate container 713f1bf1e37e
 ---> 42ef5d91553b
Step 3/7 : RUN chown testuser /home/testuser
 ---> Running in 442e8fee949f
Removing intermediate container 442e8fee949f
 ---> c12d55a9831c
Step 4/7 : RUN ls -la /home/testuser
 ---> Running in f1e0129af7fd
total 20
drwxr-xr-x 1 testuser testuser 4096 Feb 28 16:30 .
drwxr-xr-x 1 root     root     4096 Feb 28 16:30 ..
-rw-r--r-- 1 testuser testuser  220 Apr  4  2018 .bash_logout
-rw-r--r-- 1 testuser testuser 3771 Apr  4  2018 .bashrc
-rw-r--r-- 1 testuser testuser  807 Apr  4  2018 .profile
Removing intermediate container f1e0129af7fd
 ---> 706cb90a08a2
Step 5/7 : USER testuser
 ---> Running in a093b47ab2a0
Removing intermediate container a093b47ab2a0
 ---> 56be4664d148
Step 6/7 : RUN touch /home/testuser/testfile
 ---> Running in 1b35e9f3a788
Removing intermediate container 1b35e9f3a788
 ---> 674dfc04616a
Step 7/7 : RUN ls -la /home/testuser/testfile
 ---> Running in c339d69b8061
-rw-r--r-- 1 testuser testuser 0 Feb 28 16:30 /home/testuser/testfile
Removing intermediate container c339d69b8061
 ---> e33001c2b1ac
Successfully built e33001c2b1ac

Actual Outcome
The build breaks, because the directory /home/testuser as well as all its content is still owned by root

Sending build context to Docker daemon  2.048kB
Step 1/7 : FROM docker-hub.artifact.swissre.com/ubuntu:latest
latest: Pulling from ubuntu
423ae2b273f4: Pull complete
de83a2304fa1: Pull complete
f9a83bce3af0: Pull complete
b6b53be908de: Pull complete
Digest: sha256:04d48df82c938587820d7b6006f5071dbbffceb7ca01d2814f81857c631d44df
Status: Downloaded newer image for docker-hub.artifact.swissre.com/ubuntu:latest
 ---> 72300a873c2c
Step 2/7 : RUN useradd -m testuser
 ---> Running in 5e6d664b48b9
Removing intermediate container 5e6d664b48b9
 ---> 55eabfb2eb21
Step 3/7 : RUN chown testuser /home/testuser
 ---> Running in b1edc2bd93da
Removing intermediate container b1edc2bd93da
 ---> 06285e4a100f
Step 4/7 : RUN ls -la /home/testuser
 ---> Running in f8a6b0c02012
total 20
drwxr-xr-x 1 root root 4096 Feb 24 15:39 .
drwxr-xr-x 1 root root 4096 Feb 24 15:39 ..
-rw-r--r-- 1 root root  220 Apr  4  2018 .bash_logout
-rw-r--r-- 1 root root 3771 Apr  4  2018 .bashrc
-rw-r--r-- 1 root root  807 Apr  4  2018 .profile
Removing intermediate container f8a6b0c02012
 ---> cf1d0d2a4d24
Step 5/7 : USER testuser
 ---> Running in cd2378512500
Removing intermediate container cd2378512500
 ---> bf2196e69fee
Step 6/7 : RUN touch /home/testuser/testfile
 ---> Running in 74456fd74def
touch: cannot touch '/home/testuser/testfile': Permission denied
The command '/bin/sh -c touch /home/testuser/testfile' returned a non-zero code: 1

[tianon]

I'm not actually sure whether Docker supports running inside another user namespace directly in this way, but I think the (experimental) rootless variants might be more what you're looking for: #174

[jappenzesr]

Hi Tianon, thanks for getting back. We have this setup running in a productive environment since months, and it works very well for the CI/CD use case. The only issue we have discovered so far is this one.

It looks like this was an issue even when running docker:dind out of box 1-2 years ago and has been fixed in the meantime for the standard use-case (running it without configuring userns): https://gitlab.com/gitlab-org/gitlab-runner/issues/2330. Maybe that fix could be extended to support userns?

I'll give the rootless variant another try. The last time I did so I abandoned the mission due to #184.

[jappenzesr]

Interestingly, the file ownership is correctly assigned in the intermediate container, but is then somehow lost when moving to the next layer.

Consider :

FROM ubuntu:latest
RUN useradd -m testuser && ls -la /home/testuser
RUN ls -la /home/testuser
USER testuser
RUN touch /home/testuser/testfile

Output:

$ docker build .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM ubuntu:latest
 ---> 4e5021d210f6
Step 2/5 : RUN useradd -m testuser && ls -la /home/testuser
 ---> Running in 24230f707d2f
total 20
drwxr-xr-x 2 testuser testuser 4096 Apr 16 14:09 .
drwxr-xr-x 1 root     root     4096 Apr 16 14:09 ..
-rw-r--r-- 1 testuser testuser  220 Apr  4  2018 .bash_logout
-rw-r--r-- 1 testuser testuser 3771 Apr  4  2018 .bashrc
-rw-r--r-- 1 testuser testuser  807 Apr  4  2018 .profile
Removing intermediate container 24230f707d2f
 ---> 1a0ce4de8c64
Step 3/5 : RUN ls -la /home/testuser
 ---> Running in c6261f26b749
total 20
drwxr-xr-x 2 root root 4096 Apr 16 14:09 .
drwxr-xr-x 1 root root 4096 Apr 16 14:09 ..
-rw-r--r-- 1 root root  220 Apr  4  2018 .bash_logout
-rw-r--r-- 1 root root 3771 Apr  4  2018 .bashrc
-rw-r--r-- 1 root root  807 Apr  4  2018 .profile
Removing intermediate container c6261f26b749
 ---> c941b8b0e21e
Step 4/5 : USER testuser
 ---> Running in 09e6effe9905
Removing intermediate container 09e6effe9905
 ---> 007694150aa7
Step 5/5 : RUN touch /home/testuser/testfile
 ---> Running in b553ded96ba6
touch: cannot touch '/home/testuser/testfile': Permission denied
The command '/bin/sh -c touch /home/testuser/testfile' returned a non-zero code: 1

[tianon]

Sorry for the delay here. At this point, I think this is very unlikely to be an issue in the docker base image, and probably is something going weird in Docker itself or potentially even higher up the stack in the kernel. I'd suggest moving this issue either to https://github.com/moby/moby or https://github.com/docker/for-linux and see if you can get more information (especially since you've got a pretty minimal reproducer).

[jappenzesr]

Hi @tianon, thanks for getting back. Sure, I'd be happy with that. Will you move it?

[tianon]

Unfortunately GitHub issues can only be directly transferred between repositories in the same organization. 😞

My "moving" I really meant "recreating" although a link back to this discussion would certainly be appropriate! 😅