Docker run error: "idmapped mounts are not supported on the filesystem"

[hthcis]

Error

Running the following command returns error "idmapped mounts are not supported on the filesystem".
With another VM (centos stream 9) I had the same error. Without sysbox runtime it works.
I could not find an existing issue for this.

sudo docker run --runtime=sysbox-runc --rm -it --hostname my_cont busybox:glibc

Error mesage:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:428: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:322: ID-map mount on etc/resolv.conf caused: rootfs_init_linux.go:316: setting up ID-mapped mount on path /data/sysbox/rootfs/c9d572f364fbba085ad1f99793540baab3947259fd94d80deac501284005bd94/top/merged/etc/resolv.conf (likely means idmapped mounts are not supported on the filesystem at this path (unknown fs)) caused: Failed to set mount attr: operation not permitted: unknown.

Configuration

OS and hardware (it's a VM on Nutanix)

Operating System: AlmaLinux 9.0 (Emerald Puma)
CPE OS Name: cpe:/o:almalinux:almalinux:9::baseos
Kernel: Linux 5.14.0-70.22.1.el9_0.x86_64
Architecture: x86-64
Hardware Vendor: Nutanix

Disks

NAME FSTYPE FSVER LABEL FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 xfs 775M 24% /boot
└─sda2 LVM2_member LVM2 001
├─almalinux-root xfs 3.1G 61% /
└─almalinux-swap swap 1 [SWAP]
sdb
└─sdb1 ext4 1.0 43.8G 5% /data

sdb1 was formatted as xfs before, but I reformatted it because i got basically the same error as now, but it stated "xfs" in the message:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:428: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:322: ID-map mount on etc/resolv.conf caused: rootfs_init_linux.go:316: setting up ID-mapped mount on path /data/sysbox/rootfs/5255d15545d1a9250d68213bb9b202bfb10524738617a8c0b46a317c2aa6eca3/top/merged/etc/resolv.conf (likely means idmapped mounts are not supported on the filesystem at this path (xfs)) caused: Failed to set mount attr: operation not permitted: unknown.

Is currently only ext4 (not xfs) supported with idmapping?

Sysbox config

I created the following systemd services for sysbox-fs and sysboy-mgr (I do not use sybox as a wrapper, because I found no way to specify the system-fs mountpoint this way).

sysbox-fs --mountpoint /data/sysboxfs
sysbox-mgr --data-root /data/sysbox

So all data is on ext4 under /data, but the sysbox-fs.log shows FUSE still uses the default path "/var/lib/sysboxfs" not "/data/sysboxfs":

time="2022-08-18 11:44:22" level=info msg="Initiating sysbox-fs ..."
time="2022-08-18 11:44:22" level=info msg="Initializing with 'allow-immutable-remounts' knob disabled (default)"
time="2022-08-18 11:44:22" level=info msg="Initializing with 'allow-immutable-unmounts' knob enabled (default)"
time="2022-08-18 11:44:22" level=info msg="FUSE dir = /var/lib/sysboxfs"
time="2022-08-18 11:44:22" level=info msg="IOvec memParser elected"
time="2022-08-18 11:44:22" level=info msg="Listening on /run/sysbox/sysfs.sock"
time="2022-08-18 11:44:22" level=info msg="Ready ..."

Maybe this is the cause for the error?
I found a PR to fix this: #333
Can I somehow configure the FUSE path?

My sysbox built

I used make to build sysbox based on latest commit (5389057) of master branch (Date Tue Aug 9 19:44:46 2022 +0300).
I'm not aware of any issues during built.

Docker version

Client: Docker Engine - Community
Version: 20.10.17
API version: 1.41
Go version: go1.17.11
Git commit: 100c701
Built: Mon Jun 6 23:03:29 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.17.11
Git commit: a89b842
Built: Mon Jun 6 23:01:12 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.4
GitCommit: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
runc:
Version: 1.1.1
GitCommit: v1.1.1-0-g52de29d
docker-init:
Version: 0.19.0
GitCommit: de40ad0

(I have downgraded containerd to 1.6.4 because of this: containerd/containerd#7219)

Sysbox-mgr log

In case it is needed

time="2022-08-18 11:44:03" level=info msg="Starting ..."
time="2022-08-18 11:44:03" level=info msg="Sysbox data root: /data/sysbox"
time="2022-08-18 11:44:03" level=info msg="Operating in system container mode."
time="2022-08-18 11:44:03" level=info msg="Listening on /run/sysbox/sysmgr.sock"
time="2022-08-18 11:44:03" level=info msg="Ready ..."
time="2022-08-18 12:08:33" level=info msg="registered new container c9d572f364fb"
time="2022-08-18 12:08:34" level=info msg="unregistered container c9d572f364fb"
time="2022-08-18 12:08:34" level=info msg="released resources for container c9d572f364fb"

Questions

• Is the incorrect FUSE path the cause for the issue? How can I configure it?
• Is only ext4 (not xfs) supported with idmapping?

Thanks for the support.

[ctalledo]

Hi @hthcis, thanks for giving Sysbox a shot and for the detailed error report.

Is currently only ext4 (not xfs) supported with idmapping?

That was the case until recently, but that may have changed in recent kernels (and if so we need to update Sysbox to be aware of this).

My recommendation is that you put both the Docker and Sysbox data roots under ext4 if possible (i.e., /var/lib/docker, /var/lib/sysbox).

Also, we don't officially support AlmaLinux (i.e., we don't test on it), but some folks have made Sysbox work on it (see this recent PR for example).

I created the following systemd services for sysbox-fs and sysboy-mgr (I do not use sybox as a wrapper, because I found no way to specify the system-fs mountpoint this way).

You shouldn't need to do that; the sysbox installer creates a top level systemd service for sysbox and the sub-services for sysbox-fs and sysbox-mgr already (under /lib/systemd/system). The sysbox data root (which defaults to /var/lib/sysbox) is configured in the sysbox-mgr sub-service. See the config section in the User Guide for more info.

Is the incorrect FUSE path the cause for the issue? How can I configure it?

I don't think so; the error is pointing to the sysbox data root being on XFS:

/data/sysbox/rootfs/5255d15545d1a9250d68213bb9b202bfb10524738617a8c0b46a317c2aa6eca3/top/merged/etc/resolv.conf (likely means idmapped mounts are not supported on the filesystem at this path (xfs)) caused: Failed to set mount attr: operation not permitted: unknown.

So the solution would be to ensure the sysbox data root (which you configured to /data/sysbox is on ext4. Please give this a try and let us know how it goes.

[hthcis]

Thanks for the quick response.

You shouldn't need to do that; the sysbox installer creates a top level systemd service for sysbox and the sub-services for sysbox-fs and sysbox-mgr already (under /lib/systemd/system). The sysbox data root (which defaults to /var/lib/sysbox) is configured in the sysbox-mgr sub-service. See the config section in the User Guide for more info.

Somehow it didn't do that on my machine. Do you mean "make install"?

I don't think so; the error is pointing to the sysbox data root being on XFS

First that was the case, but later I reformatted /data to ext4. Now the error is:
(see my first quote in my initial post for the full message)

likely means idmapped mounts are not supported on the filesystem at this path (unknown fs)

So the solution would be to ensure the sysbox data root (which you configured to /data/sysbox is on ext4. Please give this a try and let us know how it goes.

Because /data is already ext4, my guess was the incorrect FUSE path is the cause.
I'll try to use the default path /var/lib/sysbox and /var/lib/sysboxfs with ext4 by using a symbolic link or bind mount.

Thanks.

[hthcis]

I installed a brand new AlmaLinux 9 VM with ext4 and default paths, but unfortunately the error is still the same.
It still states "unknown fs".

Error

likely means idmapped mounts are not supported on the filesystem at this path (unknown fs)

EDIT:
Maybe related: dmesg shows an idmapping error. How can i enable idmapping?

VFS: idmapped mount is not enabled.

I use a different kernel (did not run "dnf update" after installation of the VM)

Operating System: AlmaLinux 9.0 (Emerald Puma)
CPE OS Name: cpe:/o:almalinux:almalinux:9::baseos
Kernel: Linux 5.14.0-70.13.1.el9_0.x86_64
Architecture: x86-64
Hardware Vendor: Nutanix

Partitions

NAME FSTYPE FSVER LABEL FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 ext4 1.0 5.2G 28% /
├─sda2 swap 1 [SWAP]
└─sda3 ext4 1.0 781.2M 13% /boot
sdb
└─sdb1 ext4 1.0 44.1G 5% /var
sr0

Docker run

sudo docker run --runtime=sysbox-runc --rm -it --hostname my_cont busybox:glibc

Error

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:428: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:322: ID-map mount on etc/resolv.conf caused: rootfs_init_linux.go:316: setting up ID-mapped mount on path /var/lib/sysbox/rootfs/7154eaa7e2d5f0763c0cb487dc6c0abdf94676f0d3641ad76db10828ffed69ee/top/merged/etc/resolv.conf (likely means idmapped mounts are not supported on the filesystem at this path (unknown fs)) caused: Failed to set mount attr: operation not permitted: unknown.

sysbox-mgr.log

time="2022-08-19 15:49:36" level=info msg="Starting ..."
time="2022-08-19 15:49:36" level=info msg="Sysbox data root: /var/lib/sysbox"
time="2022-08-19 15:49:36" level=info msg="Operating in system container mode."
time="2022-08-19 15:49:36" level=info msg="Listening on /run/sysbox/sysmgr.sock"
time="2022-08-19 15:49:36" level=info msg="Ready ..."
time="2022-08-19 15:49:36" level=debug msg="rootfsMon starting ..."
time="2022-08-19 15:50:17" level=info msg="registered new container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="Alloc(7154eaa7e2d5, 65536) = {sysbox 165536 65536}, {sysbox 165536 65536}"
time="2022-08-19 15:50:17" level=debug msg="Prep rootfs cloning for container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="k0sVolMgr: created volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="buildkitVolMgr: created volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="containerdVolMgr: created volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="dockerVolMgr: created volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="k3sVolMgr: created volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="rke2VolMgr: created volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="kubeletVolMgr: created volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="autoRemoveCheck: Docker query start for 7154eaa7e2d5"
time="2022-08-19 15:50:18" level=debug msg="Chown rootfs clone for container 7154eaa7e2d5 (165536:165536)"
time="2022-08-19 15:50:18" level=debug msg="Revert chown rootfs clone for container 7154eaa7e2d5"
time="2022-08-19 15:50:18" level=debug msg="k0sVolMgr: sync'd-out volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:18" level=debug msg="buildkitVolMgr: sync'd-out volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:18" level=debug msg="containerdVolMgr: sync'd-out volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="dockerVolMgr: sync'd-out volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="k3sVolMgr: sync'd-out volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="rke2VolMgr: sync'd-out volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="kubeletVolMgr: sync'd-out volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="added fs watch on /var/lib/docker/overlay2/9e6e8ba2c2f15facbb09f758bf0b69e33e74a4edced10f269794b911a024c51d"
time="2022-08-19 15:50:19" level=info msg="unregistered container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="autoRemoveCheck: done for 7154eaa7e2d5 (autoRemove = true)"
time="2022-08-19 15:50:19" level=debug msg="rootfsMon: rm on /var/lib/docker/overlay2/9e6e8ba2c2f15facbb09f758bf0b69e33e74a4edced10f269794b911a024c51d"
time="2022-08-19 15:50:19" level=debug msg="k0sVolMgr: destroyed volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="buildkitVolMgr: destroyed volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="containerdVolMgr: destroyed volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="dockerVolMgr: destroyed volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="k3sVolMgr: destroyed volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="rke2VolMgr: destroyed volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="kubeletVolMgr: destroyed volume for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=debug msg="Free(7154eaa7e2d5)"
time="2022-08-19 15:50:19" level=debug msg="Removing rootfs clone for container 7154eaa7e2d5"
time="2022-08-19 15:50:19" level=info msg="released resources for container 7154eaa7e2d5"

sysboy-fs.log

time="2022-08-19 15:49:32" level=info msg="Initiating sysbox-fs ..."
time="2022-08-19 15:49:32" level=info msg="Initializing with 'allow-immutable-remounts' knob disabled (default)"
time="2022-08-19 15:49:32" level=info msg="Initializing with 'allow-immutable-unmounts' knob enabled (default)"
time="2022-08-19 15:49:32" level=info msg="FUSE dir = /var/lib/sysboxfs"
time="2022-08-19 15:49:32" level=info msg="IOvec memParser elected"
time="2022-08-19 15:49:32" level=info msg="Listening on /run/sysbox/sysfs.sock"
time="2022-08-19 15:49:32" level=info msg="Ready ..."
time="2022-08-19 15:50:17" level=debug msg="Container pre-registration started: id = 7154eaa7e2d5"
time="2022-08-19 15:50:17" level=debug msg="Created fuse server for container 7154eaa7e2d5f0763c0cb487dc6c0abdf94676f0d3641ad76db10828ffed69ee"
time="2022-08-19 15:50:17" level=info msg="Container pre-registration completed: id = 7154eaa7e2d5"
2022/08/19 15:50:17 FUSE: <- Getattr [ID=0x4 Node=0x1 Uid=0 Gid=0 Pid=1765] 0x0 fl=0
time="2022-08-19 15:50:17" level=debug msg="Requested Attr() operation for entry /"
2022/08/19 15:50:17 FUSE: -> [ID=0x4] Getattr valid=0s ino=2 size=4096 mode=drw------- uid=0 gid=0
2022/08/19 15:50:17 FUSE: <- Lookup [ID=0x6 Node=0x1 Uid=0 Gid=0 Pid=1765] "sys"
time="2022-08-19 15:50:17" level=debug msg="Requested Lookup() operation for entry sys (req ID=0x6)"
time="2022-08-19 15:50:17" level=debug msg="Executing Lookup() for req-id: 0x6, handler: root, resource: sys"
...
time="2022-08-19 15:50:18" level=debug msg="Requested Attr() operation for entry /"
2022/08/19 15:50:18 FUSE: -> [ID=0xc2] Getattr valid=0s ino=2 size=4096 mode=drw------- uid=0 gid=0
2022/08/19 15:50:18 FUSE: <- Getattr [ID=0xc4 Node=0xc Uid=0 Gid=0 Pid=1810] 0x0 fl=0
time="2022-08-19 15:50:18" level=debug msg="Requested Attr() operation for entry /proc"
2022/08/19 15:50:18 FUSE: -> [ID=0xc4] Getattr valid=0s ino=1 size=0 mode=dr-xr-xr-x uid=0 gid=0
2022/08/19 15:50:18 FUSE: <- Getattr [ID=0xc6 Node=0xf Uid=0 Gid=0 Pid=1810] 0x0 fl=0
time="2022-08-19 15:50:18" level=debug msg="Requested Attr() operation for entry /proc/uptime"
2022/08/19 15:50:18 FUSE: -> [ID=0xc6] Getattr valid=0s ino=3309900216374434016 size=0 mode=-r--r--r-- uid=0 gid=0
time="2022-08-19 15:50:18" level=debug msg="Container unregistration started: id = 7154eaa7e2d5"
time="2022-08-19 15:50:18" level=debug msg="Destroyed fuse server for container 7154eaa7e2d5f0763c0cb487dc6c0abdf94676f0d3641ad76db10828ffed69ee"
time="2022-08-19 15:50:18" level=info msg="Container unregistration completed: id = 7154eaa7e2d5"

[brauner]

I'm the maintainer of this upstream. Some CentOS versions have disabled Idmapped mounts. Last I checked @giuseppe had filed a request to enable them.

[ctalledo]

Thanks for chiming in @brauner, and thank you very much for developing the kernel's ID-mapped-mounts in the first place.

@hthcis: per @brauner's feedback and the dmesg warning VFS: idmapped mount is not enabled seems like indeed ID-mapped-mounts are not supported on your kernel unfortunately.

For Sysbox to work properly, the Linux kernel must either support ID-mapped-mounts or support the shiftfs module (works on Ubuntu/Debian mainly), as these mechanisms allow host files mounted into the Sysbox container (which is always inside a Linux user-namespace) to show up with proper user-ID/group-ID ownership inside the container.

[hthcis]

Thanks @brauner, @ctalledo for the help.
I'll test with Ubuntu.

Can I follow the status of @giuseppe's request somewhere?
Also I'm curious what kernel @roussidis used to get it to work with AlmaLinux 9. In case someone knows...

[johnwmail]

+1

[roussidis]

Sorry for my late response @hthcis and @johnwmail

When I did the tests i was using Almalinux 9.0 with kernel-5.14.0-70.17.1.el9_0.x86_64 and kernel-5.14.0-70.26.1.el9_0.x86_64

I also have tested it with Almalinux 9.1 and kernel-5.14.0-162.6.1.el9_1.x86_64.

As a suggestion i don't know if you have tried to start sysbox without idmapped-mounts enabled (it starts with them by default). I haven't seen that in the thread being suggested.

To do that start sysbox with sysbox --disable-idmapped-mount.

[johnwmail]

Sorry for my late response @hthcis and @johnwmail

When I did the tests i was using Almalinux 9.0 with kernel-5.14.0-70.17.1.el9_0.x86_64 and kernel-5.14.0-70.26.1.el9_0.x86_64

I also have tested it with Almalinux 9.1 and kernel-5.14.0-162.6.1.el9_1.x86_64.

As a suggestion i don't know if you have tried to start sysbox without idmapped-mounts enabled (it starts with them by default). I haven't seen that in the thread being suggested.

To do that start sysbox with sysbox --disable-idmapped-mount.

Nice, thanks for sharing, will try on tomorrow

[hthcis]

Thanks for the response.
We switched to Ubuntu. It worked great until they removed support in their kernel (see #596 (comment)). So for now we use Ubuntu with an older kernel version. I do not know if the newest kernel still has this issue.

[johnwmail]

Yes, sysbox --disable-idmapped-mount works on rhel9 kernel (5.14.0-162.6.1.el9_1.x86_64)

[ctalledo]

Hi folks, closing this ticket as Sysbox v0.6.2 no longer requires shiftfs on kernels >= 5.12; see here for more info.

Also, Sysbox v0.6.2 checks if shiftfs is present and functional on the host; if it's present but not working correctly, it uses alternative methods to ensure the rootfs has proper ownership inside the rootless container.