sys/devices/virtual/dmi/id/product_uuid: permission denied in ubuntu multipass environment

[andreasvh-conceto]

Hello,

currently we are facing the issue, that sysbox is not running in an ubuntu multipass environment. We can not start any container with sysbox runtime.
Error:

docker run --runtime=sysbox-runc --rm -it --hostname my_cont registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:425: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:66: setting up rootfs mounts caused: open sys/devices/virtual/dmi/id/product_uuid: permission denied: unknown.

I saw a similiar issue 405 here which seemed to be fixed inside issue 439. On my machine in the multipass environment the product id is also not set. So ls -l /sys/devices/virtual/dmi/id/product_uuid gives me nothing. Inserting it manually is not possible - also permission denied.

How to reproduce?

1. Install multipass:
   for macos
   brew install --cask multipass
   for linux see here

2. Launch a docker multipass instance (adjust the mem or disk storage as you wish for this instance):
   multipass launch docker --disk 40G --mem 8G --cpus 4 --name ubuntu

3. Access the shell
   multipass shell ubuntu

4. install sysbox:

ARCH=$(dpkg --print-architecture)

mkdir -p sysbox && cd sysbox
wget -L -O sysbox.deb https://downloads.nestybox.com/sysbox/releases/v0.5.2/sysbox-ce_0.5.2-0.linux_${ARCH}.deb
docker rm $(docker ps -a -q) -f
sudo apt install -y jq
sudo apt install -y ./sysbox.deb

sudo systemctl status sysbox -n20

rm sysbox.deb

5. ctrl+d and leave the multipass instancel
6. restart the multipass instance and access the shell again:

multipass restart ubuntu
multipass shell ubuntu

7. Run any container also simple alpine images are not working. I tried also
   docker run --runtime=sysbox-runc --rm -it --hostname my_cont registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker

Expected behaviour: Container starts without error
Acutal behaviour: Running into error:
sys/devices/virtual/dmi/id/product_uuid: permission denied: unknown.

My multipass machine specs:

~ multipass info ubuntu
Name:           ubuntu
State:          Running
IPv4:           192.168.64.6
                172.20.0.1
Release:        Ubuntu 22.04.1 LTS
Image hash:     8a0477adcbda (Ubuntu 22.04 LTS)
Load:           0.16 0.08 0.01
Disk usage:     3.8G out of 38.6G
Memory usage:   224.6M out of 7.7G
Mounts:         --

My machine specs:
Darwin MacBook-Pro.fritz.box 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103 arm64

My multipass isntance specs:

Any ideas for a workaround?

Thanks and best regards
Andreas

[rodnymolina]

@andreasvh-conceto, thanks for the detailed description.

As you pointed out, we fixed this issue a couple of months ago as part of #439, but unfortunately we haven't published a new release containing these changes yet. You can either build Sysbox from sources, or alternatively, I could provide you with a pre-release image (please login to our slack channel for the logistics).

[andreasvh-conceto]

Hi @rodnymolina

thanks for your quick reply. We tried to build the sysbox by doing the following in the multipass environment (arm architecture):

git clone --recursive https://github.com/nestybox/sysbox.git
cd sysbox
sudo apt install make
make sysbox
sudo make install
sudo ./scr/docker-cfg --sysbox-runtime=enable

restarted whole multipass instance
leave the shell
multipass restart ubuntu

sudo ./scr/sysbox

// tried out both
docker run --runtime=sysbox-runc --rm -it --hostname my_cont debian:latest
docker run --runtime=sysbox-runc hello-world

Error:

ubuntu@ubuntu:~/sysboxdev/sysbox$ docker run --runtime=sysbox-runc hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
7050e35b49f5: Pull complete
Digest: sha256:faa03e786c97f07ef34423fccceeec2398ec8a5759259f94d99078f264e9d7af
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:428: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:366: failed to mkdirall /var/lib/sysbox/shiftfs/d371ef9c-4238-43a2-a28a-201cd6b2d8e6/var/lib/rancher/rke2: mkdir /var/lib/sysbox/shiftfs/d371ef9c-4238-43a2-a28a-201cd6b2d8e6/var: value too large for defined data type caused: mkdir /var/lib/sysbox/shiftfs/d371ef9c-4238-43a2-a28a-201cd6b2d8e6/var: value too large for defined data type: unknown.

mkdircommand seems to fail. Any idea?

Thanks again for ideas.

[ctalledo]

Hi @andreasvh-conceto, that error looks like issue 596; what Linux kernel version does the host have?

[andreasvh-conceto]

Hi @ctalledo

thx for the referenced issue.

kernel version is:
ubuntu@ubuntu:~$ uname -r
5.15.0-56-generic

As mentioned above it is a multipass instance with the docker template. We did not update the kernel.

:)

[ctalledo]

Hi @andreasvh-conceto, it's likely the same issue then (shiftfs is broken in Ubuntu with kernel 5.15.0-56).

The work-around is to configure Sysbox to not use shiftfs, by configuring the systemd service unit for the sysbox-mgr (/lib/systemd/system/sysbox-mgr.service) and passing in the --disable-shiftfs flag to it. See here for some more info on how to reconfigure sysbox.

[rodnymolina]

@andreasvh-conceto, you're reproducing issue #596, I forgot to mention that yesterday (I reproduced it in my multipass setup too). Just change your /etc/systemd/system/sysbox.service.wants/sysbox-mgr.service file like this and it should all work:

ExecStart=/usr/bin/sysbox-mgr --disable-shiftfs

[andreasvh-conceto]

@ctalledo @rodnymolina thanks for your workaround. The file /etc/systemd/system/sysbox.service.wants/sysbox-mgr.service was not present, but adjusting the /lib/systemd/system/sysbox-mgr.service helped.

Are there any plans for an official release in the next time related to the initial already fixed issue? Would be great :)

Thanks a lot!

[rodnymolina]

@andreasvh-conceto, yes, we are planning to release v0.6.0 in February.

Will close this issue now. Please let us know if have any other question.