podman failed to remove a container created with buildah

[sstosh]

Issue Description

podman failed to remove a container created with buildah and prints an error message.

Error: removing storage for container "<container name>": unmounting "<graphroot>/overlay/<layer id>/merged": invalid argument

This issue is reproduced after applying a commit c83589a (#18835).
The root cause seems to be a containers/common or containers/storage repository.

Steps to reproduce the issue

latest main branch: Failed

# podman version
Client:       Podman Engine
Version:      4.7.0-dev
API Version:  4.7.0-dev
Go Version:   go1.20.6
Git Commit:   ebf368dd692c498b1e921be0dc546fb3ad50548a
Built:        Thu Aug  3 17:07:18 2023
OS/Arch:      linux/amd64

# buildah from quay.io/libpod/testimage:20221018
Trying to pull quay.io/libpod/testimage:20221018...
Getting image source signatures
Copying blob 578f06cc66c5 done
Copying blob a3ed95caeb02 done
Copying config f5a99120db done
Writing manifest to image destination
testimage-working-container

# podman rm -f testimage-working-container
WARN[0000] Unmounting container "testimage-working-container" while attempting to delete storage: unmounting "/var/lib/containers/storage/overlay/2b1d7e0f744eccac26db50d73725324c10e221014fc1a018b1061419c61be8f9/merged": invalid argument
Error: removing storage for container "testimage-working-container": unmounting "/var/lib/containers/storage/overlay/2b1d7e0f744eccac26db50d73725324c10e221014fc1a018b1061419c61be8f9/merged": invalid argument

commit c83589a: Failed

# podman version
Client:       Podman Engine
Version:      4.6.0-dev
API Version:  4.6.0-dev
Go Version:   go1.20.6
Git Commit:   c83589a8529de213b17dc69b9c7ad3e5ec86494a
Built:        Thu Aug  3 16:13:34 2023
OS/Arch:      linux/amd64

# buildah from quay.io/libpod/testimage:20221018
Trying to pull quay.io/libpod/testimage:20221018...
Getting image source signatures
Copying blob a3ed95caeb02 done
Copying blob 578f06cc66c5 done
Copying config f5a99120db done
Writing manifest to image destination
testimage-working-container

# podman rm -f testimage-working-container
WARN[0000] Unmounting container "testimage-working-container" while attempting to delete storage: unmounting "/var/lib/containers/storage/overlay/7f8faf07d665232b80b8f1e10cbaa29b90f5dd7b7bfb4f84a6a77c256776cf3b/merged": invalid argument
Error: removing storage for container "testimage-working-container": unmounting "/var/lib/containers/storage/overlay/7f8faf07d665232b80b8f1e10cbaa29b90f5dd7b7bfb4f84a6a77c256776cf3b/merged": invalid argument

commit decbab9: Succeeded

# podman version
Client:       Podman Engine
Version:      4.6.0-dev
API Version:  4.6.0-dev
Go Version:   go1.20.6
Git Commit:   decbab9223a22e9dba6918ce0a908215ae8b7cbe
Built:        Thu Aug  3 16:19:52 2023
OS/Arch:      linux/amd64

# buildah from quay.io/libpod/testimage:20221018
Trying to pull quay.io/libpod/testimage:20221018...
Getting image source signatures
Copying blob a3ed95caeb02 done
Copying blob 578f06cc66c5 done
Copying config f5a99120db done
Writing manifest to image destination
testimage-working-container

# podman rm -f testimage-working-container
testimage-working-container

Describe the results you received

# podman rm -f testimage-working-container
WARN[0000] Unmounting container "testimage-working-container" while attempting to delete storage: unmounting "/var/lib/containers/storage/overlay/7f8faf07d665232b80b8f1e10cbaa29b90f5dd7b7bfb4f84a6a77c256776cf3b/merged": invalid argument
Error: removing storage for container "testimage-working-container": unmounting "/var/lib/containers/storage/overlay/7f8faf07d665232b80b8f1e10cbaa29b90f5dd7b7bfb4f84a6a77c256776cf3b/merged": invalid argument

Describe the results you expected

# podman rm -f testimage-working-container
testimage-working-container

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.9
    systemPercent: 0.06
    userPercent: 0.04
  cpus: 36
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: server
    version: "38"
  eventLogger: journald
  freeLocks: 2048
  hostname: fedora38
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.2.9-300.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 7655833600
  memTotal: 8301731840
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc38.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc38.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8301572096
  swapTotal: 8301572096
  uptime: 0h 8m 19.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 16039018496
  graphRootUsed: 10907877376
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.0-dev
  Built: 1691050038
  BuiltTime: Thu Aug  3 17:07:18 2023
  GitCommit: ebf368dd692c498b1e921be0dc546fb3ad50548a
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0-dev

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

KVM

Additional information

It may be related to the following issue.
#18874 (comment)

#  bin/podman rm -f 335f
WARN[0000] Unmounting container "335f" while attempting to delete storage: unmounting "/var/lib/containers/storage/overlay/e3149559a6e1ae65a7b0b2e140ceb2bb7e7bdc06b9654bd5d46ba5d3adf7ad46/merged": invalid argument 
Error: removing storage for container "335f": unmounting "/var/lib/containers/storage/overlay/e3149559a6e1ae65a7b0b2e140ceb2bb7e7bdc06b9654bd5d46ba5d3adf7ad46/merged": invalid argument

[vrothberg]

@nalind, this looks like a storage issue, can you take a look?

[hedayat]

Had the same issue. I guess the problem was that I've shutdown the system, so there were nothing mounted in those dirs. rmdiring all overlay/*/merged dirs fixed it for me.

[vrothberg]

@giuseppe do you know what's happening?

[flouthoc]

I have seen this issue before and I followed same process i.e rmdir of overlay/*/merged , I think c/storage/Put( should handle such cases with rmdir internally when Unmount fails for Put. I'll open a PR for this and we can discuss there.

[edsantiago]

This is almost certainly the same as #18831, except I don't know why in Cirrus that leaves the entire system unusable.

[flouthoc]

@sstosh I think this is long standing issue and it surfaced after containers/storage#1607 where-as before I think issue was still there but it was getting ignored.