RUN chmod 777 on /var/run yields differing permissions; Kaniko vs Docker engine build

[bitsofinfo]

Dockerfile built w/ docker 19.03.4 yields an image who has these expected permissions:

The Dockerfile does a RUN chmod 777 /var/run

Contents of /var in image build by Docker 19.03.4

bash-5.0$ ls -al /var
total 44
drwxr-xr-x 1 root root 4096 Jan 23 14:36 .
drwxr-xr-x 1 root root 4096 Mar 11 22:24 ..
drwxr-xr-x 1 root root 4096 Jan 23 19:29 cache
dr-xr-xr-x 2 root root 4096 Jan 23 14:36 empty
drwxr-xr-x 1 root root 4096 Mar 10 15:51 lib
drwxr-xr-x 2 root root 4096 Jan 23 14:36 local
drwxr-xr-x 3 root root 4096 Jan 23 14:36 lock
drwxr-xr-x 1 root root 4096 Mar 10 15:51 log
drwxr-xr-x 2 root root 4096 Jan 23 14:36 opt
lrwxrwxrwx 1 root root    4 Jan 23 14:36 run -> /run
drwxr-xr-x 3 root root 4096 Jan 23 14:36 spool
drwxrwxrwt 2 root root 4096 Jan 23 14:36 tmp

bash-5.0$ ls -al /
...
drwxrwxrwx   1 root root  4096 Jan 23 14:36 run
...

Image built w/ kaniko 0.15.x or 0.18.0:

Contents of /var

bash-5.0$ ls -al /var
total 44
drwxr-xr-x 1 root root 4096 Mar 11 16:23 .
drwxr-xr-x 1 root root 4096 Mar 11 22:26 ..
drwxr-xr-x 1 root root 4096 Mar 11 16:23 cache
dr-xr-xr-x 2 root root 4096 Jan 23 14:36 empty
drwxr-xr-x 1 root root 4096 Mar 11 16:23 lib
drwxr-xr-x 2 root root 4096 Jan 23 14:36 local
drwxr-xr-x 3 root root 4096 Jan 23 14:36 lock
drwxr-xr-x 1 root root 4096 Mar 11 16:23 log
drwxr-xr-x 2 root root 4096 Jan 23 14:36 opt
lrwxrwxrwx 1 root root    4 Jan 23 14:36 run -> /run
drwxr-xr-x 3 root root 4096 Jan 23 14:36 spool
drwxrwxrwt 2 root root 4096 Jan 23 14:36 tmp

bash-5.0$ ls -al /
...
drwxr-xr-x   2 root root  4096 Jan 23 14:36 run
...

My app breaks because it runs as non-root, the statement in the Dockerfile RUN chmod 777 /var/run is intended to set this.

Why does it work on Docker engine builds but not kaniko?

Kaniko perms:

drwxr-xr-x   2 root root  4096 Jan 23 14:36 run

Docker built perms:

drwxrwxrwx   1 root root  4096 Jan 23 14:36 run

[tejal29]

looks like, this issue is specific to symlinks.
Looks like the contents of /var has right perimissions.

The second output screenshot which you provided, is this for /var/run or /run

drwxr-xr-x   2 root root  4096 Jan 23 14:36 run

[bitsofinfo]

Correct, its symlink related. That is /run and when built by docker ce the perms are 777, w/ kaniko they are 755

[tejal29]

@bitsofinfo please provide your dockerfile.
I want to verify if the base image already has /run or it is created by kaniko.

/cc @tstromberg
Thanks
Tejal

[bitsofinfo]

Unfortunately I cannot share the dockerfiles as-is as it derives from other private images etc.

[bitsofinfo]

@tejal29 @dani29 I just verified that the base image this dockerfile is FROM, already has both /run and the symlink /var/run that points to it. (pre-existing)

ultimately it derives from nginx:1.17.8-alpine

[dani29]

Hi @bitsofinfo,

Did you use the flag --whitelist-var-run=false in your build command? (see #1011 for more info)
I couldn't build the following Dockerfile without explicitly setting it to false, but when I did, the following image was built with the correct permissions on the target file.

FROM nginx:1.17.8-alpine
RUN ls -la /
RUN ls -la /var

RUN chmod 777 /var/run

RUN ls -la /
RUN ls -la /var

[bitsofinfo]

Thanks.

I did not use that flag. So you are saying that flag is required to do what I'm trying to do? Why would this not just work by default?

[dani29]

I believe that by default (=without setting the flag to false), /var/run will not be part of the file-system snapshot taken after some of the commands. There's some explanation in fs_util.go, but perhaps @tejal29 could elaborate on why this is required. Intuitively, I agree that setting "whitelist = false" for a directory you do want to include is confusing...

Specifically in your case, without seeing the Dockerfile it's hard to guess at what stage /var/run "reappears" in the file-system. I can just say that the sample Dockerfile I attached couldn't be build with kaniko because RUN chmod 777 /var/run failed, because this directory was filtered earlier.

Does adding the flag produces correct build?

[dani29]

@bitsofinfo After a further research, I wanted to provide a small clarification: the flag --whitelist-var-run=false is required only if you'd like to unpack the /var/run directory from the base image. In your case, since you did not use this flag, this directory would not be unpacked from the base. However, it might have been re-created in later stages if you run other commands like apt-get install, and potentially with wrong permissions.

In any case, seems like running chmod on the symlink works on an arbitrary image. Please let me know if I can help in any way.

[tejal29]

@bitsofinfo I verified your dockerfile on the latest build with --whitelist-var-run=false flag like @dani29 mentioned.

/ # /kaniko/executor -f Dockerfile --context=dir://workspace --destination=gcr.io/tejal-test/test1 --whitelist-var-run=false
INFO[0000] Retrieving image manifest nginx:1.17.8-alpine 
INFO[0001] Retrieving image manifest nginx:1.17.8-alpine 
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest nginx:1.17.8-alpine 
INFO[0003] Retrieving image manifest nginx:1.17.8-alpine 
INFO[0003] Executing 0 build triggers                   
INFO[0003] Unpacking rootfs as cmd RUN ls -la / requires it. 
INFO[0004] RUN ls -la /                                 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] Resolving 4673 paths                         
INFO[0005] cmd: /bin/sh                                 
INFO[0005] args: [-c ls -la /]                          
INFO[0005] Running: [/bin/sh -c ls -la /]               
total 84
drwxr-xr-x    1 root     root          4096 May  4 18:04 .
drwxr-xr-x    1 root     root          4096 May  4 18:04 ..
-rwxr-xr-x    1 root     root             0 May  4 18:03 .dockerenv
drwxr-xr-x    2 root     root          4096 May  4 18:04 bin
drwxr-xr-x    2 root     root         12288 May  4 18:03 busybox
drwxr-xr-x    5 root     root           360 May  4 18:03 dev
drwxr-xr-x    1 root     root          4096 May  4 18:04 etc
drwxr-xr-x    2 root     root          4096 May  4 18:04 home
drwxr-x---    6 407936   89939         4096 May  4 18:04 kaniko
drwxr-xr-x    5 root     root          4096 May  4 18:04 lib
drwxr-xr-x    5 root     root          4096 May  4 18:04 media
drwxr-xr-x    2 root     root          4096 May  4 18:04 mnt
drwxr-xr-x    2 root     root          4096 May  4 18:04 opt
dr-xr-xr-x  397 root     root             0 May  4 18:03 proc
drwx------    3 root     root          4096 May  4 18:04 root
drwxr-xr-x    2 root     root          4096 May  4 18:04 run
drwxr-xr-x    2 root     root          4096 May  4 18:04 sbin
drwxr-xr-x    2 root     root          4096 May  4 18:04 srv
dr-xr-xr-x   13 root     root             0 May  4 18:03 sys
drwxrwxrwt    2 root     root          4096 May  4 18:04 tmp
drwxr-xr-x    7 root     root          4096 May  4 18:04 usr
drwxr-xr-x   11 root     root          4096 May  4 18:04 var
drwxr-x---    6 407936   89939         4096 May  4 18:03 workspace
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] Resolving 4673 paths                         
INFO[0005] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0005] RUN ls -la /var                              
INFO[0005] cmd: /bin/sh                                 
INFO[0005] args: [-c ls -la /var]                       
INFO[0005] Running: [/bin/sh -c ls -la /var]            
total 44
drwxr-xr-x   11 root     root          4096 May  4 18:04 .
drwxr-xr-x    1 root     root          4096 May  4 18:04 ..
drwxr-xr-x    5 root     root          4096 May  4 18:04 cache
dr-xr-xr-x    2 root     root          4096 May  4 18:04 empty
drwxr-xr-x    5 root     root          4096 May  4 18:04 lib
drwxr-xr-x    2 root     root          4096 May  4 18:04 local
drwxr-xr-x    3 root     root          4096 May  4 18:04 lock
drwxr-xr-x    3 root     root          4096 May  4 18:04 log
drwxr-xr-x    2 root     root          4096 May  4 18:04 opt
lrwxrwxrwx    1 root     root             4 May  4 18:04 run -> /run
drwxr-xr-x    3 root     root          4096 May  4 18:04 spool
drwxrwxrwt    2 root     root          4096 May  4 18:04 tmp
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] Resolving 4673 paths                         
INFO[0006] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0006] RUN chmod 777 /var/run                       
INFO[0006] cmd: /bin/sh                                 
INFO[0006] args: [-c chmod 777 /var/run]                
INFO[0006] Running: [/bin/sh -c chmod 777 /var/run]     
INFO[0006] Taking snapshot of full filesystem...        
INFO[0006] Resolving 4673 paths                         
INFO[0006] RUN ls -la /                                 
INFO[0006] cmd: /bin/sh                                 
INFO[0006] args: [-c ls -la /]                          
INFO[0006] Running: [/bin/sh -c ls -la /]               
total 84
drwxr-xr-x    1 root     root          4096 May  4 18:04 .
drwxr-xr-x    1 root     root          4096 May  4 18:04 ..
-rwxr-xr-x    1 root     root             0 May  4 18:03 .dockerenv
drwxr-xr-x    2 root     root          4096 May  4 18:04 bin
drwxr-xr-x    2 root     root         12288 May  4 18:03 busybox
drwxr-xr-x    5 root     root           360 May  4 18:03 dev
drwxr-xr-x    1 root     root          4096 May  4 18:04 etc
drwxr-xr-x    2 root     root          4096 May  4 18:04 home
drwxr-x---    6 407936   89939         4096 May  4 18:05 kaniko
drwxr-xr-x    5 root     root          4096 May  4 18:04 lib
drwxr-xr-x    5 root     root          4096 May  4 18:04 media
drwxr-xr-x    2 root     root          4096 May  4 18:04 mnt
drwxr-xr-x    2 root     root          4096 May  4 18:04 opt
dr-xr-xr-x  394 root     root             0 May  4 18:03 proc
drwx------    3 root     root          4096 May  4 18:04 root
drwxrwxrwx    2 root     root          4096 May  4 18:04 run
drwxr-xr-x    2 root     root          4096 May  4 18:04 sbin
drwxr-xr-x    2 root     root          4096 May  4 18:04 srv
dr-xr-xr-x   13 root     root             0 May  4 18:03 sys
drwxrwxrwt    2 root     root          4096 May  4 18:04 tmp
drwxr-xr-x    7 root     root          4096 May  4 18:04 usr
drwxr-xr-x   11 root     root          4096 May  4 18:04 var
drwxr-x---    6 407936   89939         4096 May  4 18:03 workspace
INFO[0006] Taking snapshot of full filesystem...        
INFO[0006] Resolving 4673 paths                         
INFO[0006] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0006] RUN ls -la /var                              
INFO[0006] cmd: /bin/sh                                 
INFO[0006] args: [-c ls -la /var]                       
INFO[0006] Running: [/bin/sh -c ls -la /var]            
total 44
drwxr-xr-x   11 root     root          4096 May  4 18:04 .
drwxr-xr-x    1 root     root          4096 May  4 18:04 ..
drwxr-xr-x    5 root     root          4096 May  4 18:04 cache
dr-xr-xr-x    2 root     root          4096 May  4 18:04 empty
drwxr-xr-x    5 root     root          4096 May  4 18:04 lib
drwxr-xr-x    2 root     root          4096 May  4 18:04 local
drwxr-xr-x    3 root     root          4096 May  4 18:04 lock
drwxr-xr-x    3 root     root          4096 May  4 18:04 log
drwxr-xr-x    2 root     root          4096 May  4 18:04 opt
lrwxrwxrwx    1 root     root             4 May  4 18:04 run -> /run
drwxr-xr-x    3 root     root          4096 May  4 18:04 spool
drwxrwxrwt    2 root     root          4096 May  4 18:04 tmp
INFO[0006] Taking snapshot of full filesystem...        
INFO[0006] Resolving 4673 paths                         
INFO[0007] No files were changed, appending empty layer to config. No layer added to image. 
/ # ls -al /var/run/
total 8
drwxrwxrwx    2 root     root          4096 May  4 18:04 .
drwxr-xr-x    1 root     root          4096 May  4 18:04 ..
/ # 

The build is successful and the permissions are right.
I am going to close this now. Please re-open if you see this issue