Podman unmounts `$graphroot/overlay` on each run

[intelfx]

Issue Description

With the default overlay driver, if the user configuration involves mounting <graphroot>/overlay (e.g., /var/lib/containers/storage/overlay) from a separate location, then podman will silently unmount this directory on each run without emitting any warning or error:

# findmnt --submounts /var/lib/containers
TARGET                                SOURCE                                     FSTYPE OPTIONS
/var/lib/containers                   rpool/SCRATCH/cain/containers/root         zfs    rw,noatime,xattr,posixacl,casesensitive
└─/var/lib/containers/storage/overlay rpool/SCRATCH/cain/containers/root/overlay zfs    rw,noatime,xattr,posixacl,casesensitive

# podman system check
podman system check  6,80s user 6,60s system 109% cpu 12,199 total

# findmnt --submounts /var/lib/containers
TARGET              SOURCE                             FSTYPE OPTIONS
/var/lib/containers rpool/SCRATCH/cain/containers/root zfs    rw,noatime,xattr,posixacl,casesensitive

# podman system check
Damaged layer 5e3e0330c68fc47581b1694a37fb68633d15e1fd459190628bb81afb3ccb454c:
2 errors occurred:
        * layer 5e3e0330c68fc47581b1694a37fb68633d15e1fd459190628bb81afb3ccb454c: creating file-getter: readlink /var/lib/containers/storage/overlay/5e3e0330c68fc47581b1694a37fb68633d15e1fd459190628bb81afb3ccb454c/diff: no such file or directory
        * layer 5e3e0330c68fc47581b1694a37fb68633d15e1fd459190628bb81afb3ccb454c: faccessat /var/lib/containers/storage/overlay/5e3e0330c68fc47581b1694a37fb68633d15e1fd459190628bb81afb3ccb454c: no such file or directory

<...>

Error: damage detected in local storage

Steps to reproduce the issue

Steps to reproduce the issue

1. Setup an empty /var/lib/containers, configure storage.conf to use overlay driver
2. Mount /var/lib/containers/storage/overlay from a separate location
3. Run podman pull ... to populate the storage
4. Observe /var/lib/containers/storage/overlay not mounted anymore
5. Run podman system check and observe errors

Describe the results you received

$graphroot/overlay is unmounted by podman after each command; subsequent podman system check reports integrity errors because everything in the storage directory has disappeared.

Describe the results you expected

Podman is able to work correctly in a configuration where $graphroot/overlay has been mounted by the user.

podman info output

# podman info
host:
  arch: amd64
  buildahVersion: 1.41.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  - dmem
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.13-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b'
  cpuUtilization:
    idlePercent: 90.3
    systemPercent: 5.65
    userPercent: 4.05
  cpus: 22
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2048
  hostname: cain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.15.11-arch0pf7-3-my
  linkmode: dynamic
  logDriver: journald
  memFree: 30541389824
  memTotal: 66831835136
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1
    path: /usr/lib/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.23.1-1
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2025_08_05.309eefd-1
    version: |
      pasta 2025_08_05.309eefd
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 68719472640
  swapTotal: 68719472640
  uptime: 0h 20m 25.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1153921974272
  graphRootUsed: 9961472
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 15
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.6.1
  Built: 1757054398
  BuiltTime: Fri Sep  5 08:39:58 2025
  GitCommit: 1e2b2315150b2ffa0971596fb5da8cd83f3ce0e1
  GoVersion: go1.24.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[mheon]

@giuseppe PTAL

[intelfx]

Please see the cross-linked PR for a quick&dirty attempt at a fix (more for illustration purposes than anything else).

[giuseppe]

Why would the user mount that directory manually?

It is an internal mount point, why do you need to touch it?

[intelfx]

@giuseppe

Why would the user mount that directory manually?

The reason to do this is to punt the container images (== conceptually discardable data) onto a storage medium with different characteristics than the rest of the graphroot.

(E.g.: compressed, deduplicated, less redundant, etc.)

Can you suggest any alternatives?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.