ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": setting up the process: open libpod/tmp/pause.pid: no such file or directory

[bill-scales]

Issue Description

Rootless podman nested inside rootless podman running as root user is creating libpod/tmp/pause.pid in current working directory, if the working directory is then changed further invocations of podman fail because it cannot find this file

Steps to reproduce the issue

Simplified steps to show the problem. /etc/subuid and /etc/subgid are not setup correctly in this example, and the launch of the outer container is missing flags such as --security-opt label=disable --device /dev/fuse which are required to run nested rooted containers

Note issue only occurs when running as user root in the container. If the container is started with --user , or you switch to another user inside the container then running nested podman commands works as expected.

1. Outer podman version (probably irrelevant, but included for completeness)

podman --version
podman version 4.6.1

2. Start rootless container

podman run -it --rm quay.io/podman/stable:latest /bin/bash

3. Inner podman version

[root@f51a0bb05b3e /]# podman --version
podman version 4.9.4

4. From CWD = dir1 run "podman image ls"

[root@f51a0bb05b3e /]# mkdir ~/dir1 && cd ~/dir1
[root@f51a0bb05b3e dir1]# podman image ls
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user
REPOSITORY TAG IMAGE ID CREATED SIZE

5. Unexpectedly podman has created files in the current working directory

[root@f51a0bb05b3e dir1]# ls -R
.:
libpod

./libpod:
tmp

./libpod/tmp:
pause.pid

6. From CWD = dir2 run "podman image ls"

[root@f51a0bb05b3e dir1]# mkdir ~/dir2 && cd ~/dir2
[root@f51a0bb05b3e dir2]# podman image ls
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user
error creating temporary file: No such file or directory
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": setting up the process: open libpod/tmp/pause.pid: no such file or directory

Describe the results you received

1st invocation of podman created libpod/tmp/pause.pid in current working directroy

Subsequent invocation of podman from a different current working directory then failed to find this file:

ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": setting up the process: open libpod/tmp/pause.pid: no such file or directory

Describe the results you expected

Expected podman to create libpod/tmp/pause.pid in a location which could be found by subsequent commands - perhaps /run/libpod/tmp/pause.pid or $XDG_RUNTIME_DIR/libpod/tmp/pause.pid

podman info output

Output from podman info run inside the 1st container@

[root@a63a36220497 /]# podman info
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user 
host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 96.17
    systemPercent: 0.94
    userPercent: 2.9
  cpus: 64
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: file
  freeLocks: 2048
  hostname: a63a36220497
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-513.18.1.el8_9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 178586238976
  memTotal: 540150956032
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.x86_64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 763641856
  swapTotal: 4294963200
  uptime: 466h 4m 19.00s (Approximately 19.42 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.fc39.x86_64
      Version: |-
        fusermount3 version: 3.16.1
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.1
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 27377394933760
  graphRootUsed: 20065934868480
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.9.4
  Built: 1711445992
  BuiltTime: Tue Mar 26 09:39:52 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4

[root@a63a36220497 /]#

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Also tested a container running a development version of podman 5.1 and reproduced the same issue, so fairly confident that this bug hasn't been fixed in newer releases.

Using strace you can see podman creating libpod, libpod/tmp and libpod/tmp/pause.pid with relative pathnames the first time it is run:

#strace -f podman image ls 2>&1 | grep libpod
...
[pid 130] newfstatat(AT_FDCWD, "libpod/tmp", 0xc000649e48, 0) = -1 ENOENT (No such file or directory)
[pid 130] newfstatat(AT_FDCWD, "libpod", 0xc000649f18, 0) = -1 ENOENT (No such file or directory)
[pid 130] mkdirat(AT_FDCWD, "libpod", 0700 <unfinished ...>
[pid 130] mkdirat(AT_FDCWD, "libpod/tmp", 0700 <unfinished ...>
[pid 133] openat(AT_FDCWD, "libpod/tmp/pause.pid.vexbwk", O_RDWR|O_CREAT|O_EXCL, 0600 <unfinished ...>
[pid 133] renameat2(AT_FDCWD, "libpod/tmp/pause.pid.vexbwk", AT_FDCWD, "libpod/tmp/pause.pid", RENAME_NOREPLACE <unfinished ...>
...

Running a second time from a different directory you can see podman fail to find the file using a relative pathname:

#strace -f podman image ls 2>&1 | grep libpod
...
[pid 251] newfstatat(AT_FDCWD, "libpod/tmp/pause.pid", 0xc00050b6f8, 0) = -1 ENOENT (No such file or directory)
[pid 261] openat(AT_FDCWD, "libpod/tmp/pause.pid.x7LxQA", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 ENOENT (No such file or directory)
[pid 251] openat(AT_FDCWD, "libpod/tmp/pause.pid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 251] write(2, "time="2024-04-10T07:32:31Z" leve"..., 212time="2024-04-10T07:32:31Z" level=error msg="invalid internal status, try resetting the pause process with "podman system migrate": setting up the process: open libpod/tmp/pause.pid: no such file or directory"

Appears to be a bug in how podman is constructing the pathname for this file.

[giuseppe]

opened a PR: #22608