podman: mount of /sys/devices/virtual/dmi/id/product_name failed

[kaikli]

What happened:

When trying to create a new cluster with podman a error occures during creation of the kind-control-plane container.
I also tried to set Delegate=yes, because of #2872.

~ ❯ systemd-run --setenv=KIND_EXPERIMENTAL_PROVIDER=podman --scope --user kind create cluster 
Running scope as unit: run-r8b434b85abeb4ec4b5888f3172c7b888.scope
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✗ Preparing nodes 📦  
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"
systemd-run --setenv=KIND_EXPERIMENTAL_PROVIDER=podman --scope --user kind

Found the mount error in podman logs kind-control-plane

~ ❯ podman logs kind-control-plane
INFO: running in a user namespace (experimental)
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
mount: /sys: permission denied.
INFO: UserNS: ignoring mount fail
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
mount: /sys/devices/virtual/dmi/id/product_name: filesystem was mounted, but any subsequent operation failed: Unknown error 5005.

What you expected to happen: Cluster created successfully

How to reproduce it (as minimally and precisely as possible): run systemd-run --setenv=KIND_EXPERIMENTAL_PROVIDER=podman --scope --user kind create cluster on Arch Linux with podman

Anything else we need to know?:

kind export logs: kind-export-logs.tar.gz

Environment:

• kind version: (use kind version): kind v0.17.0 go1.20.1 linux/amd64
• Runtime info: (use docker info or podman info): see kind-export-logs.tar.gz
• OS (e.g. from /etc/os-release): Arch Linux
• Kubernetes version: (use kubectl version): v1.26.1
• Any proxies or other special environment settings?: no

[aojea]

Unknown error 5005

what is this error

INFO: remounting /sys read-only
mount: /sys: permission denied.

can this be relevant @AkihiroSuda ?

[BenTheElder]

I would recommend running Kubernetes rootfully for the moment.

We're just doing a bind mount here, if that's failing I'm not sure how much of anything else will succeed ...

[kaikli]

Just tried it again with the new kind version v0.22.0.

Still getting the same error but found the following error in dmesg. I thought maybe this could help.

[ 2570.721720] podman2: port 1(veth0) entered blocking state
[ 2570.721758] podman2: port 1(veth0) entered disabled state
[ 2570.721778] veth0: entered allmulticast mode
[ 2570.721835] veth0: entered promiscuous mode
[ 2570.723122] podman2: port 1(veth0) entered blocking state
[ 2570.723128] podman2: port 1(veth0) entered forwarding state
[ 2571.549046] overlayfs: failed to set xattr on upper
[ 2571.549051] overlayfs: ...falling back to redirect_dir=nofollow.
[ 2571.549053] overlayfs: ...falling back to metacopy=off.
[ 2571.549054] overlayfs: ...falling back to index=off.
[ 2571.549055] overlayfs: ...falling back to uuid=null.
[ 2571.549057] overlayfs: ...falling back to xino=off.
[ 2571.549058] overlayfs: try mounting with 'userxattr' option
[ 2571.670835] podman2: port 1(veth0) entered disabled state
[ 2571.671469] veth0 (unregistering): left allmulticast mode
[ 2571.671472] veth0 (unregistering): left promiscuous mode
[ 2571.671474] podman2: port 1(veth0) entered disabled state

When I create a new image with podman commit and run the mount inside a container with this image I get the following error but no error is shown in dmesg.

root@3da977f1ff6b:/# mount -o ro,bind /kind/product_name /sys/class/dmi/id/product_name
mount: /sys/devices/virtual/dmi/id/product_name: mount point is not a directory.
       dmesg(1) may have more information after failed mount system call.