open sys/devices/virtual/dmi/id/product_uuid: permission denied

[myugan]

Hi, I'm facing an issue when executing the container using sysbox (0.4.0) here is the log that I got using docker inspect

OCI runtime create failed: container_linux.go:393: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:68: setting up rootfs mounts caused: open sys/devices/virtual/dmi/id/product_uuid: permission denied: unknown

[rodnymolina]

Hi @myugan, can you please paste here the instruction you are using to launch the container? Also, if you're using docker, can you share the /etc/docker/daemon.json configuration?

Also, please provide the Linux distro and kernel.

[myugan]

This is my config file:

{
    "debug": true,
    "log-level": "info",
    "features": {
        "buildkit": true
    },
    "hosts": [
        "unix:///var/run/docker.sock",
        "tcp://0.0.0.0:2375"
    ],
    "tls": true,
    "tlscacert": "/etc/docker/certs/ca.pem",
    "tlscert": "/etc/docker/certs/server-cert.pem",
    "tlskey": "/etc/docker/certs/server-key.pem",
    "tlsverify": true,
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/bin/sysbox-runc"
        }
    },
    "userns-remap": "sysbox",
    "default-runtime": "sysbox-runc"
}

Kernel: Linux host 5.4.0-80-generic #90~18.04.1-Ubuntu SMP Tue Jul 13 19:40:02 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Distro: Ubuntu 18.04.5 LTS

[rodnymolina]

I can reproduce it neither with v0.4.1 nor with v0.4.0. Please provide the following:

• the docker instruction you're using to launch the sys-container
• docker info
• docker version

[myugan]

docker version output:

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:40 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.8
  GitCommit:        7eba5930496d9bbe375fdf71603e610ad737d2b2
 sysbox-runc:
  Version:          :                   0.4.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info output:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 16
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge cilium host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc sysbox-runc
 Default Runtime: sysbox-runc
 Init Binary: docker-init
 containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc version:
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-80-generic
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.64GiB
 Name: labs-staging
 ID: YBYV:4BEZ:536T:3J7S:HPA5:ZAIW:MTSR:WAKV:VDTI:BFJC:L4ZJ:C2PT
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 112
  Goroutines: 125
  System Time: 2021-10-06T03:57:01.697355094Z
  EventsListeners: 2
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

[myugan]

Previously I tried to upgrade to 0.4.1 but got the same error and rollback the version to 0.4.0 seems it affected the previous version

[rodnymolina]

Looks like your docker config (/etc/docker/daemon.json) and docker process are not in sync. Notice that you have userns attribute in docker config but not in docker info output. I would expect something like this in your output:

$ docker info | grep userns
WARNING: No swap limit support
  userns
$

Did you restart dockerd after making those config changes?

[myugan]

Yes, i restarted it also reboot the server but still got the same problem

[rodnymolina]

That's strange. Can you double-check that docker is really running in your machine (systemctl status docker)? It would be the first time that I see userns-remap configured without being reflected in docker info output.

[ctalledo]

If it helps, I am using Docker 20.10.9 in userns-remap mode, and docker info shows this:

root@sysbox-test:~/nestybox# docker info | grep userns
WARNING: No swap limit support
  userns

[rodnymolina]

@myugan, can you please answer my previous question above?

If you are unable to reproduce the issue due to this one being a non-sysbox problem, please close this one when have a chance.

[myugan]

Closing this issue, since I restored the machine with the previous backup.

[mkarimim]

I have the same issue, It's because there's no product_uuid in /sys/devices/virtual/dmi/id/ address so container could not be run and print this error

root@ppwp2:/home/user# docker run --runtime=sysbox-runc -it ubuntu:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:393: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:68: setting up rootfs mounts caused: open sys/devices/virtual/dmi/id/product_uuid: permission denied: unknown.
ERRO[0000] error waiting for container: context canceled 

by running sudo dmidecode -t system -t baseboard command, I got UUID: Not Present:

root@ppwp2:/home/user# sudo dmidecode -t system -t baseboard
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.

Handle 0x0001, DMI type 1, 27 bytes
System Information
	Manufacturer: MSI
	Product Name: MS-7816
	Version: 1.0
	Serial Number: To be filled by O.E.M.
	UUID: Not Present
	Wake-up Type: Power Switch
	SKU Number: To be filled by O.E.M.
	Family: To be filled by O.E.M.

by normal server I got the correct uuid:

root@hzwp1:/home/user# sudo dmidecode -t system -t baseboard
sudo: unable to resolve host hzwp1: Name or service not known
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x0001, DMI type 1, 27 bytes
System Information
	Manufacturer: FUJITSU
	Product Name:  
	Version:  
	Serial Number:  
	UUID: xxxx3098-xxxx-9546-xxxx-deadxxxx6563
	Wake-up Type: Other
	SKU Number:  
	Family: ESPRIMO-FTS

and /sys/devices/virtual/dmi/id/product_uuid exists on server.

[ctalledo]

Thanks @mkarimim. Looks like Sysbox is assuming that the host will have a valid /sys/devices/virtual/dmi/id/product_uuid, but that may not be the case in your machine.

Question: in your host, does /sys/devices/virtual/dmi/id/product_uuid exist?

ls -l /sys/devices/virtual/dmi/id/product_uuid
cat  /sys/devices/virtual/dmi/id/product_uuid

For example, in my dev machine I see:

# ls -l /sys/devices/virtual/dmi/id/product_uuid
-r-------- 1 root root 4096 Dec  9 19:34 /sys/devices/virtual/dmi/id/product_uuid

# cat /sys/devices/virtual/dmi/id/product_uuid
68bba0fe-80c4-4a36-88b2-281bb00ba9be

Thanks!

[rodnymolina]

@mkarimim, that's right, I recently noticed this error in Linode's KVM machines where VMs were being spawned without the kvm/qemu's uuid parameter, which would explain why you get UUID: Not Present message when running dmidecode.

We have a fix in mind that will eliminate this Sysbox dependency.

[kevinnls]

@rodnymolina is there a workaround that one could use in the meantime, please?

[ctalledo]

Hi @kevinnls, what do you see on your host with:

$ ls -l /sys/devices/virtual/dmi/id/product_uuid 
$ cat /sys/devices/virtual/dmi/id/product_uuid

Does that file exist? If not, then you can work-around the problem by creating a fake uuid as follows:

$ echo "1234abcd-0001-0002-0003-0123456789ab" > /root/fake_product_uuid
$ mount --bind /root/fake_product_uuid /sys/devices/virtual/dmi/id/product_uuid

You need root access to do that. Use a better / random UUID too.

[kevinnls]

hi, @ctalledo . thanks for the response

Does that file exist? [...]

no

$ echo "1234abcd-0001-0002-0003-0123456789ab" > /root/fake_product_uuid
$ mount --bind /root/fake_product_uuid /sys/devices/virtual/dmi/id/product_uuid

You need root access to do that. Use a better / random UUID too.

this is how it went ://

sudo -i
uuidgen >fakeuuid
mount -B fakeuuid /sys/devices/virtual/dmi/id/product_uuid
### ERR ### mount: /sys/devices/virtual/dmi/id/product_uuid: mount point does not exist.
touch $_
### ERR ### touch: [...] permission denied
mkdir $_
### ERR ### mkdir: [...] operation not permitted

i did check whoami and id to receive the outputs root and 0 as required

additional output:

ls -ldZ /sys/devices/virtual/dmi/id/
### OUT ### drwxr-xr-x. 3 root root system_u:object_r:sysfs_t:s0 0 Dec 12 16:01 /sys/devices/virtual/dmi/id

i am on a Linode running Fedora34

[ctalledo]

Hi @kevinnls,

ERR ### mount: /sys/devices/virtual/dmi/id/product_uuid: mount point does not exist.

I see, the file is not there and you can't even create a fake one on top. I am afraid that's the only (hacky) workaround I had in mind.

Short of that, fixing this will require that we make a change in Sysbox such that if the host machine does not have the /sys/devices/virtual/dmi/id/product_uuid file, Sysbox can deal with that and potentially fake one of those inside the containers it deploys.

Question: Sysbox works best on Ubuntu (because it carries a module called shiftfs that Sysbox uses to ensure host files mounted into the rootless Sysbox container show up with proper owner:group). Have you tried Ubuntu on Linode?

[kevinnls]

hi @ctalledo

Question: Sysbox works best on Ubuntu [...] Have you tried Ubuntu on Linode?

no.

i'll give it a shot later and share the results on this thread

[rodnymolina]

@kevinnls, please keep in mind that the same problem should be present in Ubuntu. As I said above, the problem here is a consequence of how Linode instantiates its KVM-based VMs, so the issue will be seen in all the Linux distros installed in those VMs.

Btw, we haven't seen this issue in any other cloud vendor so far.

We should have a proper fix for this one in the next release (in ~ 3 weeks from now).

[rodnymolina]

@kevinnls, if you can't wait till the next release, please ping us in our slack channel so that we can discuss other options ...

[kevinnls]

...

Question: Sysbox works best on Ubuntu [...] Have you tried Ubuntu on Linode?

no.
i'll give it a shot later and share the results on this thread

as @rodnymolina noted

[...] the issue will be seen in all the Linux distros installed in those VMs. [...]

same results on Ubuntu 20.04 on Linode. the product_uuid file does not exist and cannot be created

[ctalledo]

Thanks @kevinnls for confirming. Looks like a work-around is not possible then, we will need the work-around in Sysbox itself.

[ctalledo]

Duplicate of sysbox issue #439. Let's use that one to track the issue.