Recursive chown is really slow

[katsar0v]

• This is a bug report
• [] This is a feature request
• I searched existing issues before opening this one (found similar issue, which was closed, but the problem still persist with me)

Expected behavior

I want to build an image with simple wordpress and theme installation (with nginx, mysql and so on).

Actual behavior

I am building an image with WordPress and own templates. In order for everything to work correctly, I need to set chown www-data:www-data /var/www/html/ -R. Unfortunately this takes long time, at least 5 minutes most of the time.

Steps to reproduce the behavior

Step 91/95 : WORKDIR /var/www/html/
 ---> Running in 3bbf283b7604
Removing intermediate container 3bbf283b7604
 ---> abe7aa3891b3
Step 92/95 : RUN chown www-data:www-data . -R

This is where this hangs for at least 5+ minutes.

Output of docker version:

Docker version 18.06.0-ce, build 0ffa825

Output of docker info:

Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 2246
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.0-46-generic
Operating System: Ubuntu 17.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.688GiB
Name: katsarov-ThinkPad-T470s
ID: R5S6:67UI:QUZK:YBZS:VXLQ:YEFK:SEGU:7EAO:WTUZ:63R5:V3UX:NEJT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

None, running on a fresh ubuntu machine.

[cpuguy83]

This is likely because every file has to be copied up on a CoW filesystem. This also means data is duplicated in the image.

How many/what size are these files? Can you add them in with the correct uid/gid instead of chowning in the build? (Note there is a "--chown" flag on the COPY instruction)

[katsar0v]

Hi, thank for your reply. I will try the --chown flag. The chown is executed on a wordpress installation (not big, but many files). But what if the files come from git, not from host machine?

[ZelphirKaltstahl]

I noticed the --chown COPY argument does not seem to support variables defined inside the Dockerfile. I presume it is because those are only defined at build time?

This is important when you do not know the user ahead of time and have it stored in a variable or when you do not wish to hardcode the user into many places inside the Dockerfile, so that you do not need to change so many places when changing the user / username.

Should this be a new issue?

[Grant1219]

I also started running into the issue today while debugging a container that is running chown -R over about 40k files and about 400MB of files. It takes roughly 12 minutes to complete.

The more interesting part is that this same container takes only 2 minutes to start up on an Ubuntu 14.04 server (4.4 kernel), but with Ubuntu 16.04 (4.15 kernel) the time required increases significantly. Docker versions on both servers are the same 18.03.1-ce.

EDIT:
After more digging, I discovered the underlying storage driver changed from "aufs" to "overlay2" so I believe the slowdown must be because of that change.

[cpuguy83]

The good news, in the new kernel overlay does a meta-data only copy for this instead of copying the whole file.

[Grant1219]

@cpuguy83 Which version will that be included in?

[cpuguy83]

@Grant1219 Looks like 4.19 -- https://cateee.net/lkddb/web-lkddb/OVERLAY_FS_METACOPY.html
You can set the default as a kernel config as well as set a mount option.

[katsar0v]

Does switching to a newer kernel really help in this case?

[cpuguy83]

If you switch and turn on meta-only copy by default in the kernel config, I think it would make a huge difference, but I have not tested yet.

Also keep in mind this is only available in 4.19 which is still in the RC phase.

[cpuguy83]

The smaller the sizes and greater number of files the less effective it would be, I suspect.
But I don't see how chown could improve beyond that change.

[kavehv]

I just migrated from my Ubuntu 14.04 VM to an 18.04 VM and attempted to bring up the same docker image in a container on the same version of docker (18.06.1). We also do a similar chown in our entrypoint. It's FAR slow in 18.04 than it was in 14.04. Both are set up to use overlay2.

[katsar0v]

Okay, this seems to be real issue and not a small one. @kavehv you say it gets worse on 18.04, can you also compare the kernel versions?

[kavehv]

14.04:

$ docker info
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 147
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
Kernel Version: 4.4.0-79-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 10.35GiB
Name: kaveh-ubuntu
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

18.04:

$ docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 33
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-34-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 10.47GiB
Name: kaveh-ubuntu
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[kavehv]

Definitely seems like a consistent problem with 18.04 and overlay2. We see it on more than one system here. Is this the right project to report this issue?

[cpuguy83]

Probably best to report to Canonical in this case.