shiftfs issues when calling sysbox-runc on unbuntu 22.04 fresh installation

[PMooreWand]

I've had to do a rebuild my workstation recently with ubuntu 20.04 using sysbox 0.5.2 and hit an issue with the mkdirall under the /var/lib/sysbox:
Docker daemon is set up for rootless fine and will create containers when not calling the sysbox-runc

 phil@phil-XPS-15-9500:~$ docker run --runtime=sysbox-runc hello-world
 docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:425: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:366: failed to mkdirall /var/lib/sysbox/shiftfs/570eae26-9d31-42c1-ab58-00c777ce17dc/var/lib/k0s: mkdir /var/lib/sysbox/shiftfs/570eae26-9d31-42c1-ab58-00c777ce17dc/var: value too large for defined data type caused: mkdir /var/lib/sysbox/shiftfs/570eae26-9d31-42c1-ab58-00c777ce17dc/var: value too large for defined data type: unknown.
 ERRO[0000] error waiting for container: context canceled 

Without sysbox-runc called.

phil@phil-XPS-15-9500:~$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Versions below,

 phil@phil-XPS-15-9500:~/.docker$ uname -a
Linux phil-XPS-15-9500 5.15.0-48-generic #54~20.04.1-Ubuntu SMP Thu Sep 1 16:17:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

 phil@phil-XPS-15-9500:~/.docker$ sysbox-mgr --version
 sysbox-mgr
 	edition: 	Community Edition (CE)
 	version: 	0.5.2
 	commit: 	ea1b7db91031355cb10b850125e0d6502dc38962
 	built at: 	Wed May 18 19:49:36 UTC 2022
 	built by: 	Rodny Molina
 	
 phil@phil-XPS-15-9500:~/.docker$ docker version 
 Client: Docker Engine - Community
  Version:           20.10.18
  API version:       1.41
  Go version:        go1.18.6
  Git commit:        b40c2f6
  Built:             Thu Sep  8 23:11:45 2022
  OS/Arch:           linux/amd64
  Context:           default
  Experimental:      true
 
 Server: Docker Engine - Community
  Engine:
   Version:          20.10.18
   API version:      1.41 (minimum version 1.12)
   Go version:       go1.18.6
   Git commit:       e42327a
   Built:            Thu Sep  8 23:09:37 2022
   OS/Arch:          linux/amd64
   Experimental:     false
  containerd:
   Version:          1.6.8
   GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
  runc:
   Version:          1.1.4
   GitCommit:        v1.1.4-0-g5fd4c4d
  docker-init:
   Version:          0.19.0
   GitCommit:        de40ad0

appears to create full set of dirs except shiftfs one in /var/lib/sysbox:

phil@phil-XPS-15-9500:/var/lib$ sudo tree sysbox
sysbox
├── buildkit
│   └── 86bf0f5e28b159543a23fe9bc6e987d2305e0b8aa20233b17891ccc6697e482a
├── containerd
│   └── 86bf0f5e28b159543a23fe9bc6e987d2305e0b8aa20233b17891ccc6697e482a
├── docker
│   └── 86bf0f5e28b159543a23fe9bc6e987d2305e0b8aa20233b17891ccc6697e482a
├── k0s
│   └── 86bf0f5e28b159543a23fe9bc6e987d2305e0b8aa20233b17891ccc6697e482a
├── kubelet
│   └── 86bf0f5e28b159543a23fe9bc6e987d2305e0b8aa20233b17891ccc6697e482a
├── rancher-k3s
│   └── 86bf0f5e28b159543a23fe9bc6e987d2305e0b8aa20233b17891ccc6697e482a
├── rancher-rke2
│   └── 86bf0f5e28b159543a23fe9bc6e987d2305e0b8aa20233b17891ccc6697e482a
└── shiftfs

Shiftfs is loaded in the kernel by the looks of it :

phil@phil-XPS-15-9500:/etc/systemd/system/multi-user.target.wants$ lsmod | grep shiftfs
shiftfs                28672  0

core.17208.tar.gz

[ctalledo]

Hi @PMooreWand,

mkdir /var/lib/sysbox/shiftfs/570eae26-9d31-42c1-ab58-00c777ce17dc/var: value too large for defined data type: unknown.

That error typically means there is a problem with shiftfs; I've seen it before when shiftfs is not build from a branch that matches the kernel.

Questions:

1. Did you build the shiftfs module yourself (from shiftfs-dkms)?

2. What's your kernel version (uname -r)?

3. If the answer to (1) is "Yes", what shiftfs-dkms branch did you use?

[PMooreWand]

hey @ctalledo,
Yes I used the shifs from shiftfs-dkms
Uname -r is 5.15.0-48-generic so used branch K5.16 following the table on master there.

k5.16	5.15.x (longterm version) (and probably 5.16.x)	-

k5.16 5.15.x (longterm version) (and probably 5.16.x) -

[PMooreWand]

Just to add to this, I did the shiftfs build after my kernel provided one hit the issue previously:

phil@phil-XPS-15-9500:~/Downloads$ modinfo shiftfs
filename:       /lib/modules/5.15.0-48-generic/kernel/fs/shiftfs.ko
license:        GPL v2
description:    id shifting filesystem
author:         Christian Brauner <christian.brauner@ubuntu.com>
author:         Seth Forshee <seth.forshee@canonical.com>
author:         James Bottomley
alias:          fs-shiftfs
srcversion:     B0C2D82DE327B38F653B659
depends:        
retpoline:      Y
intree:         Y
name:           shiftfs
vermagic:       5.15.0-48-generic SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        71:BE:D6:27:24:EE:C7:44:45:7A:53:20:37:5C:FD:A2:10:6B:68:79
sig_hashalgo:   sha512
signature:      44:B5:80:8E:F1:FA:53:36:16:D8:9F:17:4E:F4:CA:4A:CE:C6:95:30:
		21:E1:A8:7C:A1:7F:82:F1:EA:88:B2:E8:FE:A5:DE:09:C8:22:12:2F:
		DF:FA:B0:8C:72:A6:BC:76:72:31:FD:66:CE:D9:8F:95:05:0B:88:62:
		C8:AA:D2:D3:01:19:CC:96:A0:48:CD:8D:F2:15:21:83:6C:56:92:73:
		BA:85:18:E1:07:99:09:88:03:C6:4F:D9:31:D1:79:18:57:83:65:F9:
		FA:F4:F9:2B:05:A2:02:01:A7:7A:3F:FA:83:FC:2B:35:37:77:6C:DF:
		44:76:2C:02:93:BC:D7:FC:7D:C1:59:F2:D8:A0:EE:EE:DF:EC:28:20:
		48:6C:34:C4:17:5D:D1:D2:CE:14:2B:E0:06:E5:89:C2:83:B5:98:EE:
		31:2A:CF:8B:4B:CF:A3:8D:7F:90:EC:BD:99:9C:13:CD:14:D8:ED:A6:
		A1:AD:FE:1E:09:C4:57:E0:11:DB:85:AE:B8:28:6D:DB:B7:9A:28:07:
		4F:AF:40:7D:53:C0:9E:00:D7:73:C6:7C:24:76:8D:05:3C:FB:E5:35:
		9D:CA:B0:83:60:85:BE:23:7A:EC:96:A4:2D:D6:AB:23:FD:83:39:F5:
		23:59:88:00:D2:C8:F1:90:62:D1:23:13:BA:E5:21:B6:8D:CF:EF:F1:
		95:C8:31:C3:AA:20:D6:A6:1B:0B:2E:D8:DD:0D:4B:CF:C8:75:9F:F2:
		6E:90:2F:CF:B4:E4:82:29:E1:53:73:32:07:EE:E0:C4:E5:8B:4E:A1:
		BB:B3:77:F1:8C:21:0B:EE:7F:6B:4F:AB:38:23:AA:C3:81:EA:E4:83:
		4B:EA:28:F3:04:79:D3:0D:3F:AB:D2:8A:F8:EB:1B:6F:BB:0B:CD:E7:
		FD:F4:B8:FE:63:75:4F:82:0A:A1:48:E5:6A:7E:DE:7C:1A:21:E7:B2:
		92:A5:80:3C:13:CD:67:C5:C9:EB:A1:CE:D1:80:F3:81:D3:FC:15:14:
		60:FA:FD:CD:97:9A:30:DC:2F:09:7D:DD:65:63:92:39:15:68:31:8D:
		6E:A6:6C:A4:5B:95:BE:D6:B4:C5:E7:60:3E:D2:07:4D:2A:3E:1E:B4:
		27:CA:B9:24:BF:60:F7:E2:65:50:61:CA:09:EE:DD:E5:2E:B2:58:5B:
		AC:D9:E0:C5:C0:7E:03:93:55:8E:1F:B8:02:3E:C5:FC:24:87:3E:B9:
		AD:E5:E3:D3:AD:A7:5D:DE:33:C4:1C:1B:30:7F:DD:E4:1D:1B:94:4B:
		1B:37:C7:4C:6E:02:FD:B7:4E:82:DF:1E:90:29:0C:60:0E:65:CC:D0:
		2F:75:32:77:1F:E9:8A:1D:B3:64:92:AA

[PMooreWand]

@ctalledo I rolled back to my previous kernel 5.15.0-46-generic and reinstalled sysbox which is now working, I'm happy enough to close this off as a bad kernel upgrade on my part and not a sysbox issue.

[ctalledo]

Sounds good @PMooreWand ; I am 100% certain that the problem is related to shiftfs not playing well with the kernel version you had, but not quite sure why. I've seen this happen when the Ubuntu kernel is missing some patches (particularly in the overlayfs filesystem) that are required for it to interact with shiftfs correctly.

Let's close for now.

[audrenbdb]

I've upgrade kernel to 5.19.0-051900-generic and it solved the problem.