Overlay on overlay fails

[ibuildthecloud]

For some reason if /var/lib/rkt is on overlay rkt fails to the with the below error

2015/10/03 20:14:07 Wrote filesystem to /var/lib/rkt/pods/run/9e234754-f188-441f-bd22-a8e528242a9b
2015/10/03 20:14:07 Writing image manifest
2015/10/03 20:14:07 error setting up app image: error rendering overlay filesystem: error writing image manifest: open /var/lib/rkt/pods/run/9e234754-f188-441f-bd22-a8e528242a9b/stage1/rootfs/opt/stage2/etcd/manifest: no such device or address

The easiest way to reproduce is run rkt in rkt with v0.8.1. I don't think this is a problem with overlay itself as I do this with Docker all the time.

[iaguis]

Apparently, mounting an overlay fs inside another one doesn't work:

# mkdir upper lower work merged
# mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged/
# cd merged
# mkdir upper2 lower2 work2 merged2
# mount -t overlay overlay -olowerdir=lower2,upperdir=upper2,workdir=work2 merged2/
# cd merged2
# echo hello > test.txt
bash: test.txt: No such device or address

I also tried to mount /var/lib/docker as an overlay fs and it doesn't work either:

# cd /var/lib/docker-overlay
# mkdir lower upper work
# mount -t overlay overlay -olowerdir=/var/lib/docker-overlay/lower,upperdir=/var/lib/docker-overlay/upper,workdir=/var/lib/docker-overlay/work /var/lib/docker
# systemctl start docker
# docker info
Containers: 0
Images: 0
Storage Driver: overlay
 Backing Filesystem: <unknown>
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.2-1-ARCH
Operating System: Arch Linux (containerized)
CPUs: 4
Total Memory: 5.731 GiB
Name: locke
ID: X22G:4KZL:RBHT:HI3E:NVX6:RGWU:FPYR:WHDK:WLC3:KA3O:B7QA:AYVW
# docker run -it busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
cfa753dfea5e: Pull complete 
d7057cb02084: Pull complete 
library/busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:16a2a52884c2a9481ed267c2d46483eac7693b813a63132368ab098a71303f8a
Status: Downloaded newer image for busybox:latest
Error response from daemon: open /var/lib/docker/overlay/9d2aa0d4fcf7b72a2db6b8346655a60412f16fa816d3904672b40269ea8bbc5b-init/merged/etc/hostname: no such device or address

[jonboulle]

Is this not just a known limitation with overlay?

On Mon, Oct 5, 2015 at 2:37 AM, Iago López Galeiras <
notifications@github.com> wrote:

Apparently, mounting an overlay fs inside another one doesn't work:

mkdir upper lower work merged

mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged/

cd merged

mkdir upper lower work merged

mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged/

cd merged

echo hello > test.txt

bash: test.txt: No such device or address

I also tried to mount /var/lib/docker as an overlay fs and it doesn't
work either:

cd /var/lib/docker-overlay

mkdir lower upper work

mount -t overlay overlay -olowerdir=/var/lib/docker-overlay/lower,upperdir=/var/lib/docker-overlay/upper,workdir=/var/lib/docker-overlay/work /var/lib/docker

systemctl start docker

docker info

Containers: 0
Images: 0
Storage Driver: overlay
Backing Filesystem:
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.2-1-ARCH
Operating System: Arch Linux (containerized)
CPUs: 4
Total Memory: 5.731 GiB
Name: locke
ID: X22G:4KZL:RBHT:HI3E:NVX6:RGWU:FPYR:WHDK:WLC3:KA3O:B7QA:AYVW

docker run -it busybox

Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
cfa753dfea5e: Pull complete
d7057cb02084: Pull complete
library/busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:16a2a52884c2a9481ed267c2d46483eac7693b813a63132368ab098a71303f8a
Status: Downloaded newer image for busybox:latest
Error response from daemon: open /var/lib/docker/overlay/9d2aa0d4fcf7b72a2db6b8346655a60412f16fa816d3904672b40269ea8bbc5b-init/merged/etc/hostname: no such device or address

—
Reply to this email directly or view it on GitHub
#1537 (comment).

[alban]

Apparently, mounting an overlay fs inside another one doesn't work:

I just tried your commands and it worked for me. I'm using Linux 4.1.6-200.fc22.x86_64.

[iaguis]

I just tried your commands and it worked for me. I'm using Linux 4.1.6-200.fc22.x86_64.

Huh? Weird.

[ibuildthecloud]

@iaguis I tried your commands on Linux 4.0 and it worked, but Linux 4.2.2 (Ubuntu 15.10) failed.

[ibuildthecloud]

@iaguis what kernel are you running, is this a regression in 4.2?

[iaguis]

@ibuildthecloud yep, I'm running 4.2.2:

> uname -a
Linux locke 4.2.2-1-ARCH #1 SMP PREEMPT Tue Sep 29 22:21:33 CEST 2015 x86_64 GNU/Linux

[iaguis]

I tried the overlay-on-overlay example in

Linux locke 4.1.9-2-lts #1 SMP Thu Oct 1 15:21:06 CEST 2015 x86_64 GNU/Linux

and it worked fine.

[alban]

I added a debug line in fs/inode.c:no_open() and the no such device or address error comes from there. I didn't see any obvious commit between v4.1 and v4.2 that would cause this bug.

[iaguis]

@alban did a git bisect, found the offending commit and reported it: http://marc.info/?l=linux-fsdevel&m=144422064220201&w=2

[alban]

@ibuildthecloud There is a kernel patch that should fix the issue ovl-fix-open-in-stacked-overlay.patch. It worked for me when testing with @iaguis' commands. Can you check if it also work in your configuration?

[alban]

@ibuildthecloud ping?

[jdknight]

FYI; fix has been pushed upstream in 4.3:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/diff/fs/overlayfs/inode.c?id=v4.3&id2=v4.2

[alban]

Great :)
I see the commit on https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=1c8a47df36d72ace8cf78eb6c228aa0f8027d3c2

[jonboulle]

should we add a note to a doc somewhere about this ?

[alban]

The kernel patch is now in the 4.2-stable tree.

[jonboulle]

\o/

On Fri, Nov 6, 2015 at 11:04 AM Alban Crequy notifications@github.com
wrote:

The kernel patch is now in the 4.2-stable tree
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
.

—
Reply to this email directly or view it on GitHub
#1537 (comment).

[s-urbaniak]

Reopening the issue since newer kernel (needs bisection) enforce restrictions on the upperdir (, and workdir) not to be on an overlayfs itself, see #2158 (comment).

I.e. overlay-over-overlay mounts on newer kernels (at least >=4.8.8) fail with the #1537 (comment) example with the folowing error message:

[18494.960175] overlayfs: filesystem on 'upper2' not supported as upperdir

[jonboulle]

@s-urbaniak perhaps a new issue for that one?

[s-urbaniak]

@jonboulle ok, I will close this then, and create a dedicated issue.