Docker.Image.mountJenkinsUser fails when Build executor runs in a container

[schnatterer]

Problem when using MavenInDocker and running maven like mvn 'install' the following error occurred:

[Pipeline] withDockerContainer
Jenkins seems to be running inside container 6a11e2323ec0b313a455baedc5b28a35dffce026656884298aba4f58149a3b14
$ docker run -t -d -u 1000:1000 -v /var/jenkins_home/workspace/ature_3_continuous_delivery-6HRXLXYING3RPQ2T3ATY6DOVMNEXY7ZFTGPCHQBMB7KHPI2V5JBQ/.jenkins/etc/passwd:/etc/passwd:ro -w /var/jenkins_home/workspace/ature_3_continuous_delivery-6HRXLXYING3RPQ2T3ATY6DOVMNEXY7ZFTGPCHQBMB7KHPI2V5JBQ --volumes-from 6a11e2323ec0b313a455baedc5b28a35dffce026656884298aba4f58149a3b14 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven:3.5.0-jdk-8
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
java.io.IOException: Failed to run image 'maven:3.5.0-jdk-8'. 
Error: docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54:
mounting 
/var/jenkins_home/workspace/ature_3_continuous_delivery-6HRXLXYING3RPQ2T3ATY6DOVMNEXY7ZFTGPCHQBMB7KHPI2V5JBQ/.jenkins/etc/passwd 
to rootfs 
/var/lib/docker/aufs/mnt/7b8aadc86829e9cbc74d8e03555a06bdd67d00865f671ac777638cd94bfe2fe9 
at 
/var/lib/docker/aufs/mnt/7b8aadc86829e9cbc74d8e03555a06bdd67d00865f671ac777638cd94bfe2fe9/etc/passwd 
caused not a directory: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.

[schnatterer]

The problem was related to this mount triggered by MavenInDocker (realized by Docker.Image.mountJenkinsUser):

 -v /var/jenkins_home/workspace/ature_3_continuous_delivery-6HRXLXYING3RPQ2T3ATY6DOVMNEXY7ZFTGPCHQBMB7KHPI2V5JBQ/.jenkins/etc/passwd:/etc/passwd:ro

The problem on this specific Jenkins instance was, that the build executor of the Jenkins instance was a docker container that mounted the docker socket so Jenkins Pipelines can start Docker containers on the same docker host as the build executor (i.e. sibling containers).

However, the path /var/lib/jenkins_home on the host was mounted into /var/jenkins_home on the build executor (see jenkinsci/docker/Dockerfile).

So, when the Pipeline (on the build executor) mounted files from the workspace to its sibling container, the docker host could not mount /var/jenkins_home because it did not exist on the host (as the files were positioned under /var/lib/jenkins_home there).

So we learned:
When build executors are Docker containers that allow pipelines to access their docker host,
the JENKINS_HOME in the build executor's container should have the same absolute path as the JENKINS_HOME on the docker host.

Otherwise in Pipelines mounting files from the workspace into docker containers created within the Pipeline will fail.

There are two solutions:

• Change the JENKINS_HOME on the docker host
• Change the JENKINS_HOME in the build executor container.