Creating pod fails (even with catatonit binary)

[MarkusSchoelzel]

/kind bug

Description

Creating pods in a podman in docker container setup fails with 4.0.x,
but works fine with 3.4.x.

(I am using self compiled static binaries, but reproduced the issue with quay.io/podman/upstream and quay.io/podman/stable)

Steps to reproduce the issue:

docker run -ti --rm --privileged --device /dev/fuse:rw quay.io/podman/upstream sh -c "sed -i -e '/^utsns=/ s/=.*/=\"private\"/' /etc/containers/containers.conf; podman pod create"

Describe the results you received:
Error: building local pause image: finding pause binary: exec: "catatonit": executable file not found in $PATH

Describe the results you expected:
pod created with id as output

Additional information you deem important (e.g. issue happens only occasionally):
Version 3.4.x works:
docker run -ti --rm --privileged --device /dev/fuse:rw quay.io/podman/stable sh -c "sed -i -e '/^utsns=/ s/=.*/=\"private\"/' /etc/containers/containers.conf; podman pod create"
Output:
a46dde323dbcb64871c855dd3dd1e9e1b6bc5de7c65ddf3593dc5100fd4c7bd2

simply adding catatonic
docker run -ti --rm --privileged --device /dev/fuse:rw quay.io/podman/upstream sh -c "curl -sLo /usr/local/bin/catatonit https://github.com/openSUSE/catatonit/releases/download/v0.1.7/catatonit.x86_64; chmod +x /usr /local/bin/catatonit; sed -i -e '/^utsns=/ s/=.*/=\"private\"/' /etc/containers/containers.conf; podman pod create"
Output:
Error: cannot set cgroup parent if not creating cgroups: invalid argument

Output of podman version:

Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.16.14
Git Commit:   7877b02aacf3e8d3d37f6283c6b8aa81688fd120
Built:        Wed Mar  2 08:12:13 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.0, commit: 4c41ec2a215c502ad6c38ea255ac1082fa1d381d'
  cpus: 256
  distribution:
    distribution: fedora
    variant: container
    version: "35"
  eventLogger: file
  hostname: fceefa3cd26c
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.13.0-28-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 468859002880
  memTotal: 811139846144
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 492h 50m 49.34s (Approximately 20.50 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.0-dev
  Built: 1646208733
  BuiltTime: Wed Mar  2 08:12:13 2022
  GitCommit: 7877b02aacf3e8d3d37f6283c6b8aa81688fd120
  GoVersion: go1.16.14
  OsArch: linux/amd64
  Version: 4.0.0-dev

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[vrothberg]

Thanks for reaching out!

Error: cannot set cgroup parent if not creating cgroups: invalid argument

@giuseppe PTAL

[giuseppe]

PR here: #13413

[MarkusSchoelzel]

PR #13413 + catatonic binary fixed it:

docker run -ti --rm --privileged --device /dev/fuse:rw upstream-pr13413 sh -c "curl -sLo /usr/local/bin/catatonit https://github.com/openSUSE/catatonit/releases/download/v0.1.7/catatonit.x86_64; chmod +x /usr/local/bin/catatonit; sed -i -e '/^utsns=/ s/=.*/=\"private\"/' /etc/containers/containers.conf; podman pod create"

-> e359cc1b1ec04bd4ec3259f1ff73f549d70eec6bb812070f1aa42fa2ce28b209

Thank you for the fix and the incredibly fast response time!