Cannot do anything within rootless podman anymore

[StiglCZ]

Issue Description

Previously, I was able to use rootless podman preety much the same way as I use rooted podman, but now when I try to exec into a container, it just outputs:

docker exec -it 81802998f3a6 /bin/bash
Error: crun: writing file /sys/fs/cgroup/user.slice/user-1001.slice/user@1001.service/user.slice/libpod-81802998f3a6e484bb1ecae0b8cad54158b642cfa43aaa1fed071085421bd0dc.scope/container/cgroup.procs: Permission denied: OCI permission denied

And when I try to create a container the result is similiar
podman run -d alpine
Error: OCI runtime error: crun: error creating systemd unit libpod-09485abbb1e422abf2d3507a3f5163fefa1a804b14d80784048bee194a63ce7c.scope: got failed

I can still do things like podman ps however, so it only seems to have effect on some commands

Steps to reproduce the issue

Steps to reproduce the issue

1. Install rocky linux
2. Update to the latest version of it
3. Try to run alpine

Describe the results you received

I have no idea how it even happened, its really weird

Describe the results you expected

For it to just create the container like previously

podman info output

podman info [11:31:57]
host:
arch: amd64
buildahVersion: 1.33.8
cgroupControllers:

• memory
• pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
  package: conmon-2.1.10-1.el9.x86_64
  path: /usr/bin/conmon
  version: 'conmon version 2.1.10, commit: 3ea3d7f99779af0fcd69ec16c211a7dc3b4efb60'
  cpuUtilization:
  idlePercent: 97.47
  systemPercent: 1.11
  userPercent: 1.43
  cpus: 8
  databaseBackend: sqlite
  distribution:
  distribution: rocky
  version: "9.4"
  eventLogger: journald
  freeLocks: 2032
  hostname: pfsc-d-01
  idMappings:
  gidmap:
  • container_id: 0
    host_id: 10
    size: 1
  • container_id: 1
    host_id: 165536
    size: 65536
    uidmap:
  • container_id: 0
    host_id: 1001
    size: 1
  • container_id: 1
    host_id: 165536
    size: 65536
    kernel: 5.14.0-427.35.1.el9_4.x86_64
    linkmode: dynamic
    logDriver: journald
    memFree: 935018496
    memTotal: 33197953024
    networkBackend: netavark
    networkBackendInfo:
    backend: netavark
    dns:
    package: aardvark-dns-1.10.0-3.el9_4.x86_64
    path: /usr/libexec/podman/aardvark-dns
    version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
    ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
    crun version 1.14.3
    commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
    rundir: /run/user/1001/crun
    spec: 1.0.0
    +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
    os: linux
    pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231204.gb86afe3-1.el9.x86_64
    version: |
    pasta 0^20231204.gb86afe3-1.el9.x86_64
    Copyright Red Hat
    GNU General Public License, version 2 or later
    https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
    security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
    serviceIsRemote: false
    slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
    slirp4netns version 1.2.3
    commit: c22fde291bb35b354e6ca44d13be181c76a0a432
    libslirp: 4.4.0
    SLIRP_CONFIG_VERSION_MAX: 3
    libseccomp: 2.5.2
    swapFree: 16800280576
    swapTotal: 16802377728
    uptime: 1029h 11m 28.00s (Approximately 42.88 days)
    variant: ""
    plugins:
    authorization: null
    log:
• k8s-file
• none
• passthrough
• journald
  network:
• bridge
• macvlan
• ipvlan
  volume:
• local
  registries:
  search:
• registry.access.redhat.com
• registry.redhat.io
• docker.io
  store:
  configFile: /home/stigl/.config/containers/storage.conf
  containerStore:
  number: 8
  paused: 0
  running: 6
  stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/stigl/.local/share/containers/storage
  graphRootAllocated: 906089058304
  graphRootUsed: 21709209600
  graphStatus:
  Backing Filesystem: xfs
  Native Overlay Diff: "true"
  Supports d_type: "true"
  Supports shifting: "false"
  Supports volatile: "true"
  Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
  number: 6
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/stigl/.local/share/containers/storage/volumes
  version:
  APIVersion: 4.9.4-rhel
  Built: 1728894259
  BuiltTime: Mon Oct 14 03:24:19 2024
  GitCommit: ""
  GoVersion: go1.21.13 (Red Hat 1.21.13-4.el9_4)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

No response

Additional information

The alpine was just an example, it happens with all images.

[Luap99]

We only support the latest release upstream so I suggest you try with the latest podman version. In any case it sounds like something with the cgroups/systemd got messed up in bad way (possibly outside of podman) so I suggest you reboot first to see if this alone clears things up.

[StiglCZ]

I will try that. How would I install the latest podman on rocky linux however, do i have to build it from source or is there an other way?

[Luap99]

There is a podman-next copr but I don't see any recent succesful epel 9 builds there as the golang version is to old.

[rhatdan]

Also looks like you might have been mucking around in your /etc/subuid and /etc/subgid?

container_id: 0
host_id: 10
size: 1
container_id: 1
host_id: 165536
size: 65536
uidmap:
container_id: 0
host_id: 1001
size: 1
container_id: 1
host_id: 165536
size: 65536

[rhatdan]

Finally Permission denied usually means either SELinux or UIDs/Permission flags being set wrong.

[StiglCZ]

Haven't messed in either of those files

[StiglCZ]

After rebooting, im getting this error:

podman exec -it 81802998f3a6 /bin/sh
Error: crun: executable file /bin/sh not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

[rhatdan]

I would try to setup a different account and try it out there. I think this account is screwed up.

[StiglCZ]

On another acc, im getting:

WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available

WARN[0000] For using systemd, you may need to log in using a user session
WARN[0000] Alternatively, you can enable lingering with: loginctl enable-linger 1000 (possibly as root)

WARN[0000] Falling back to --cgroup-manager=cgroupfs WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available

WARN[0000] For using systemd, you may need to log in using a user session
WARN[0000] Alternatively, you can enable lingering with: loginctl enable-linger 1000 (possibly as root)

WARN[0000] Falling back to --cgroup-manager=cgroupfs CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
WARN[0000] Failed to add pause process to systemd sandbox cgroup: dbus: couldn't determine address of session bus

[StiglCZ]

Ok, despite the warnings i can exec into the container. How would i fix my account without wiping any data? @rhatdan

[Luap99]

you need to properly log into a rootless user to create the system user session
https://github.com/containers/podman/blob/main/troubleshooting.md#solution-28

podman exec -it 81802998f3a6 /bin/sh
Error: crun: executable file /bin/sh not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

That means the container has no /bin/sh so if the image does not have the binary that would totally be expected

[StiglCZ]

Its alpine tho, it should 100% have it

[StiglCZ]

I also tried with bash btw