layered overlayfs mounts not working on CentOS 7

[blalor]

When I do acbuild run with a dependency that also has a dependency, the overlayfs mount fails with invalid argument. I can reproduce the problem by hand:

mount \
    -t overlay \
    -olowerdir=.acbuild/depstore-expanded/sha512-1ec8db8d4fe4ac487619810d0b69a667d748e229480e61d984d8a041d44987873ec082f2cf7d024bd6600a5e8b4c09d5da2469e79b19f2f9115f6611631c48e4/rootfs:.acbuild/depstore-expanded/sha512-197a0fd2ce2b9e4569774f8fc0d5f747e7639d5b12ebf2a1f0e2054404359ae197f264960c9dec5149d486464edb1a449a88223d67524d6ba40b5c1c86836a55/rootfs,upperdir=.acbuild/currentaci/rootfs,workdir=.acbuild/work overlay .acbuild/target

I get this in dmesg:

 overlayfs: failed to resolve '.acbuild/depstore-expanded/sha512-1ec8db8d4fe4ac487619810d0b69a667d748e229480e61d984d8a041d44987873ec082f2cf7d024bd6600a5e8b4c09d5da2469e79b19f2f9115f6611631c48e4/rootfs:.acbuild/depstore-expanded/sha512-197a0fd2ce2b9e4569774f8fc0d5f747e7639d5b12ebf2a1f0e2054404359ae197f264960c9dec5149d486464edb1a449a88223d67524d6ba40b5c1c86836a55/rootfs': -2

Not sure if this works elsewhere. I see where the kernel docs say that layered lower filesystems should work, but I'm guessing CentOS 7's kernel is too old.

uname -a: Linux localhost 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I can rkt run the dependency I've added with acbuild, so the concept works.

Maybe this'll be moot when rkt/rkt#1833 lands?

[cgonyeo]

I'm able to use acbuild run with chained dependencies with the 4.2.5 kernel, so I'm betting something was added to the kernel since 3.10 that lets this work.

Just to make sure that we're talking about the same image:

derek@rokot ~/go/src/github.com/appc/acbuild> acbuild begin
derek@rokot ~/go/src/github.com/appc/acbuild> acbuild dep add aci.gonyeo.com/test
derek@rokot ~/go/src/github.com/appc/acbuild> sudo acbuild run -- apk update
[sudo] password for derek: 
Downloading aci.gonyeo.com/test: [=============================] 253 B/253 B
Downloading aci.gonyeo.com/znc: [==============================] 6 MB/6 MB   
Downloading aci.gonyeo.com/nginx: [============================] 1.3 MB/1.3 MB 
Downloading aci.gonyeo.com/alpine: [===========================] 2.49 MB/2.49 MB
fetch http://dl-4.alpinelinux.org/alpine/v3.2/main/x86_64/APKINDEX.tar.gz
v3.2.3-139-g3ce61e0 [http://dl-4.alpinelinux.org/alpine/v3.2/main]
OK: 5292 distinct packages available
derek@rokot ~/go/src/github.com/appc/acbuild>

@jonboulle @alban is this something rkt has had to work around in the past?

[blalor]

This is what I get:

[root@localhost ~]# acbuild begin
[root@localhost ~]# acbuild dep add aci.gonyeo.com/test
[root@localhost ~]# acbuild run -- apk update
Downloading aci.gonyeo.com/test: [=============================] 253 B/253 B
Downloading aci.gonyeo.com/znc: [==============================] 6 MB/6 MB 
Downloading aci.gonyeo.com/nginx: [============================] 1.3 MB/1.3 MB 
Downloading aci.gonyeo.com/alpine: [===========================] 2.49 MB/2.49 MB
run: invalid argument

[jonboulle]

The only thing that comes to mind off the top of my head is rkt/rkt#1537

[blalor]

@dgonyeo's test now works on CentOS 7.2:

[root@gocd-agent-fc5b8a3b ~]# acbuild begin
[root@gocd-agent-fc5b8a3b ~]# acbuild dep add aci.gonyeo.com/test
[root@gocd-agent-fc5b8a3b ~]# acbuild run -- apk update
Downloading aci.gonyeo.com/test: [=============================] 253 B/253 B
Downloading aci.gonyeo.com/znc: [==============================] 5.4 MB/5.4 MB 
Downloading aci.gonyeo.com/alpine: [===========================] 2.49 MB/2.49 MB
Downloading aci.gonyeo.com/nginx: [============================] 1.3 MB/1.3 MB 
Downloading aci.gonyeo.com/alpine: [===========================] 2.49 MB/2.49 MB
fetch http://dl-4.alpinelinux.org/alpine/v3.2/main/x86_64/APKINDEX.tar.gz
v3.2.3-172-g7448d89 [http://dl-4.alpinelinux.org/alpine/v3.2/main]
OK: 5295 distinct packages available
[root@gocd-agent-fc5b8a3b ~]# uname -a
Linux gocd-agent-fc5b8a3b 3.10.0-327.4.4.el7.x86_64 #1 SMP Tue Jan 5 16:07:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@gocd-agent-fc5b8a3b ~]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 

[cgonyeo]

Weird. Nothing should have changed on our end. Maybe a heisenbug?

[blalor]

Well, my initial report was on CentOS 7.1, and it seems to be working correctly on 7.2. So, I think it's more likely an issue with the kernel that was fixed in the newer release.

[cgonyeo]

Good to know. Guess I'll close this for now then, and if it pops back up the issue can be reopened.