Container creation as root, works as rootless user but not root (podman 4.8.3, conmon 2.1.8)

[nktrmb]

Currently getting undesirable behavior when attempting to create a container from a root user, but when performing the same or similar action from a rootless user the container is created without issues. This is the same for the custom container or if the container is simply hello-world.

Error from root user: Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

podman info:

 arch: arm
 buildahVersion: 1.33.2
 cgroupControllers:
 - memory
 - pids
 cgroupManager: systemd
 cgroupVersion: v2
 conmon:
   package: Unknown
   path: /usr/bin/conmon
   version: 'conmon version 2.1.8, commit: 6d88cb3672a3dceeb4b045a92dc4d4285c9f4efd'
 cpuUtilization:
   idlePercent: 49.84
   systemPercent: 22.96
   userPercent: 27.21
 cpus: 2
 databaseBackend: sqlite
 distribution:
   codename: nanbield
   distribution: trmb-judo
   version: 0.7.0.dev0-2024.1.4
 eventLogger: journald
 freeLocks: 2047
 hostname: mp1010
 idMappings:
   gidmap: null
   uidmap: null
 kernel: 6.1.69-g-g
 linkmode: dynamic
 logDriver: journald
 memFree: 3126398976
 memTotal: 4098801664
 networkBackend: cni
 networkBackendInfo:
   backend: cni
   dns: {}
 ociRuntime:
   name: runc
   package: Unknown
   path: /usr/bin/runc
   version: |-
     runc version 1.1.10+dev
     commit: v1.1.10-2-gf3446b1e-dirty
     spec: 1.0.2-dev
     go: go1.20.13
     libseccomp: 2.5.5
 os: linux
 pasta:
   executable: ""
   package: ""
   version: ""
 remoteSocket:
   exists: true
   path: /run/podman/podman.sock
 security:
   apparmorEnabled: false
   capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
   rootless: false
   seccompEnabled: true
   seccompProfilePath: ""
   selinuxEnabled: false
 serviceIsRemote: false
 slirp4netns:
   executable: /usr/bin/slirp4netns
   package: Unknown
   version: |-
     slirp4netns version 1.2.0-beta.0+dev
     commit: unknown
     libslirp: 4.7.0
     SLIRP_CONFIG_VERSION_MAX: 4
     libseccomp: 2.5.5
 swapFree: 0
 swapTotal: 0
 uptime: 0h 1m 20.00s
 variant: v7
plugins:
 authorization: null
 log:
 - k8s-file
 - none
 - passthrough
 - journald
 network:
 - bridge
 - macvlan
 - ipvlan
 volume:
 - local
registries:
 search:
 - docker.io
 - registry.fedoraproject.org
 - quay.io
 - registry.access.redhat.com
 - registry.centos.org
store:
 configFile: /etc/containers/storage.conf
 containerStore:
   number: 5
   paused: 0
   running: 0
   stopped: 5
 graphDriverName: overlay
 graphOptions:
   overlay.mountopt: nodev
 graphRoot: /root/.local/share/containers/storage
 graphRootAllocated: 28565897216
 graphRootUsed: 1130864640
 graphStatus:
   Backing Filesystem: overlayfs
   Native Overlay Diff: "false"
   Supports d_type: "true"
   Supports shifting: "true"
   Supports volatile: "true"
   Using metacopy: "false"
 imageCopyTmpDir: /var/tmp
 imageStore:
   number: 1
 runRoot: /root/.local/share/containers/storage/temp
 transientStore: false
 volumePath: /root/.local/share/containers/storage/volumes
version:
 APIVersion: 4.8.3-dev
 Built: 1702297875
 BuiltTime: Mon Dec 11 12:31:15 2023
 GitCommit: 0ec4c8b1d7d6fc273d50064f87a6c0b2d269fdcd
 GoVersion: go1.20.13
 Os: linux
 OsArch: linux/arm
 Version: 4.8.3-dev

I also updated to 2.1.10 of conmon, and different versions of podman (4.7.3-> latest) and it was the same result. I originally had the data store locations as /var/lib/containers/storage and /run/containers/storage, (i.e. the default) but this also did not get around this error.

uname -a
Linux device-name 6.1.69-g-g #1 SMP PREEMPT Wed Feb 7 15:26:29 UTC 2024 armv7l GNU/Linux

[nktrmb]

After further review, I managed to get it to create containers (as root) but only if I downgraded the cgroup version to V1. I have similar firmware on another device (similar as in its the base yocto for the device I am using), and that works with everything at cgroup v2. Currently looking into kernel configuration options that might be necessary on the main device.

[haircommander]

do you have output from conmon to share? if you're using podman it should be in the journal

[nktrmb]

Logs from Journalctl from conmon (Used the '/' to filter the logs, and also tried grepping for 'conmon' but there were no additional logs):

Feb 27 17:28:06 mp1010 kernel: cni-podman0: port 1(veth89018c49) entered forwarding state
Feb 27 17:28:06 mp1010 systemd-networkd[551]: veth89018c49: Gained carrier
Feb 27 17:28:06 mp1010 NetworkManager[542]: <info>  [1709054886.9482] device (veth89018c49): carrier: link connected
Feb 27 17:28:06 mp1010 NetworkManager[542]: <info>  [1709054886.9493] device (cni-podman0): carrier: link connected
Feb 27 17:28:06 mp1010 systemd-networkd[551]: cni-podman0: Gained carrier
Feb 27 17:28:06 mp1010 avahi-daemon[448]: Joining mDNS multicast group on interface cni-podman0.IPv4 with address 10.88.0.1.
Feb 27 17:28:06 mp1010 avahi-daemon[448]: New relevant interface cni-podman0.IPv4 for mDNS.
Feb 27 17:28:06 mp1010 avahi-daemon[448]: Registering new address record for 10.88.0.1 on cni-podman0.IPv4.
Feb 27 17:28:07 mp1010 systemd[1]: Created slice Slice /machine.
Feb 27 17:28:07 mp1010 systemd[1]: Started libpod-conmon-ad3dabb2995145d288173985ae7c855a36cd54930afa9a840939342d678795a7.scope.
Feb 27 17:28:07 mp1010 systemd[1]: Started libcontainer container ad3dabb2995145d288173985ae7c855a36cd54930afa9a840939342d678795a7.
Feb 27 17:28:07 mp1010 systemd[1]: libpod-ad3dabb2995145d288173985ae7c855a36cd54930afa9a840939342d678795a7.scope: Deactivated successfully.
Feb 27 17:28:07 mp1010 conmon[1394]: conmon ad3dabb2995145d28817 <error>: Failed to receive console file descriptor Communication error on send
Feb 27 17:28:07 mp1010 systemd-networkd[551]: veth89018c49: Link DOWN
Feb 27 17:28:07 mp1010 systemd-networkd[551]: veth89018c49: Lost carrier
Feb 27 17:28:07 mp1010 kernel: cni-podman0: port 1(veth89018c49) entered disabled state
Feb 27 17:28:07 mp1010 kernel: device veth89018c49 left promiscuous mode
Feb 27 17:28:07 mp1010 kernel: cni-podman0: port 1(veth89018c49) entered disabled state
Feb 27 17:28:07 mp1010 systemd-networkd[551]: cni-podman0: Lost carrier
Feb 27 17:28:07 mp1010 systemd[1]: run-netns-netns\x2d08d81828\x2db6e2\x2d5b34\x2d653f\x2d907d6a0da988.mount: Deactivated successfully.
Feb 27 17:28:07 mp1010 systemd[1]: data-root-podman-containers-storage-overlay-ff40ac06a0334fef199ba31f3692b2bff4ab123fee4e8c2d0201631ca67e97c5-merged.mount: Deactivated successfully.
Feb 27 17:28:07 mp1010 systemd[1]: data-root-podman-containers-storage-overlay\x2dcontainers-ad3dabb2995145d288173985ae7c855a36cd54930afa9a840939342d678795a7-userdata-shm.mount: Deactivated successfully.
Feb 27 17:28:07 mp1010 systemd[1]: data-root-podman-containers-storage-overlay.mount: Deactivated successfully.
Feb 27 17:28:07 mp1010 systemd[1]: libpod-conmon-ad3dabb2995145d288173985ae7c855a36cd54930afa9a840939342d678795a7.scope: Deactivated successfully.
Feb 27 17:28:08 mp1010 systemd-networkd[551]: cni-podman0: Gained IPv6LL

Below is the snippet of the --log-level=debug argument logs when trying to run any container.

DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349 -u 1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349 -r /usr/bin/runc -b /data/root/podman/containers/storage/overlay-containers/1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349/userdata -p /run/containers/storage/overlay-containers/1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349/userdata/pidfile -n magical_diffie --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349 --full-attach -s -l journald --log-level debug --syslog -t --conmon-pidfile /run/containers/storage/overlay-containers/1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /data/root/podman/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /data/root/podman/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349.scope 
DEBU[0000] Cleaning up container 1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349 
DEBU[0000] Tearing down network namespace at /run/netns/netns-404f2334-04f5-e68a-6139-f79101f3f101 for container 1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349 
DEBU[0001] Unmounted container "1de7586de255062a5230fc5a3f3c5a44663b1f4419e922433f8cf87978dc2349" 
DEBU[0001] ExitCode msg: "container create failed (no logs from conmon): conmon bytes \"\": readobjectstart: expect { or n, but found \x00, error found in #0 byte of ...||..., bigger context ...||..." 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Version:

~# podman --version
podman version 5.0.2-dev
~# conmon --version
conmon version 2.1.10
commit: affab49967eb62f75d2a47398344ab053326289f

[arichiardi]

This happens to me as well with the following settings when using a rootless user that does not have any additional group on the host

conmon version 2.1.13
commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b

Mar 03 11:47:54 xyz podman[80387]: 2025-03-03 11:47:54.94975923 -0700 MST m=+0.035401095 container create a0831564a3445da21bb0474a2e7ec68ad0441b17ddfa22add612f3251b40226d (image=docker.io/nvidia/cuda:12.8.0-base-ubuntu22.04, name=llm_nvidia_test_tmp10251, io.podman.compose.config-hash=64c97fec1dde6eef4f42f77b03d41e55af78baa1466d10419c7410c27aa92271, com.docker.compose.service=nvidia_test, maintainer=NVIDIA CORPORATION <cudatools@nvidia.com>, com.docker.compose.project=llm, PODMAN_SYSTEMD_UNIT=podman-compose@llm.service, io.podman.compose.version=1.3.0, org.opencontainers.image.version=22.04, io.podman.compose.project=llm, com.docker.compose.project.working_dir=/var/lib/llm, com.docker.compose.container-number=1, com.docker.compose.project.config_files=docker-compose.yml, org.opencontainers.image.ref.name=ubuntu)

Mar 03 11:50:01 xyz conmon[80642]: conmon 4e549152c15be8d7ee84 <error>: Failed to receive console file descriptor Communication error on send
Mar 03 11:50:01 xyz podman[80617]: 2025-03-03 11:50:01.712709448 -0700 MST m=+0.013780412 image pull 0a9a35ada16c65c2a05557259c620eb43fa6041a8722a8a42d0a8882e08cfe21 docker.io/nvidia/cuda:12.8.0-base-ubuntu22.04
...

This is the message that stands out

"Failed to receive console file descriptor Communication error on send"

[jnovy]

There seem to be still issues with JSON parsing back-end even in the current code base - Inadequate error handling in recvmsg() calls - only checked for negative return values, basic JSON string escaping that didn't handle edge cases like null strings, UTF-8 characters, or all control characters and no fallback mechanism when JSON generation fails. I will do a PR to improve this.