Error: copying system image from manifest list: trying to reuse blob */diff: no such file or directory

[luckylinux]

Issue Description

I am facing a very weird issue.

My standard folder structure is like this (the idea behind splitting into several folders was to make it easier to handle e.g. ZFS based snapshots and backups):

/home/podman/bin
/home/podman/build
/home/podman/cache
/home/podman/certificates
/home/podman/compose
/home/podman/config
/home/podman/containers
/home/podman/data
/home/podman/images
/home/podman/local
/home/podman/log
/home/podman/root
/home/podman/run
/home/podman/secrets
/home/podman/storage
/home/podman/tmp
/home/podman/volumes

The issue does NOT show up with ZFS. Everything seems to work fine there, with zdata/PODMAN/ mounted with --rbind to /home/podman/:

zdata/PODMAN on /zdata/PODMAN type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/BUILD on /zdata/PODMAN/BUILD type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/CACHE on /zdata/PODMAN/CACHE type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/DATA on /zdata/PODMAN/DATA type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/LOG on /zdata/PODMAN/LOG type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/STORAGE on /zdata/PODMAN/STORAGE type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/IMAGES on /zdata/PODMAN/IMAGES type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/CONFIG on /zdata/PODMAN/CONFIG type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/COMPOSE on /zdata/PODMAN/COMPOSE type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/CERTIFICATES on /zdata/PODMAN/CERTIFICATES type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/VOLUMES on /zdata/PODMAN/VOLUMES type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/SECRETS on /zdata/PODMAN/SECRETS type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/LOCAL on /zdata/PODMAN/LOCAL type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/ROOT on /zdata/PODMAN/ROOT type zfs (rw,noatime,xattr,noacl,casesensitive)
zdata/PODMAN/CONFIG on /home/podman/.config/containers type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/STORAGE on /home/podman/storage type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/BUILD on /home/podman/build type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/CERTIFICATES on /home/podman/certificates type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/COMPOSE on /home/podman/compose type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/CONFIG on /home/podman/config type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/LOG on /home/podman/log type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/ROOT on /home/podman/root type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/DATA on /home/podman/data type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/IMAGES on /home/podman/images type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/VOLUMES on /home/podman/volumes type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/CACHE on /home/podman/cache type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/LOCAL on /home/podman/local type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)
zdata/PODMAN/SECRETS on /home/podman/secrets type zfs (rw,noatime,xattr,noacl,casesensitive,x-systemd.automount)

The issue appears only on EXT4.
Here, no mount --rbind is used, and the folders are just plain folders within the user home directory.

/home/podman/bin
/home/podman/build
/home/podman/cache
/home/podman/certificates
/home/podman/compose
/home/podman/config
/home/podman/containers
/home/podman/data
/home/podman/images
/home/podman/local
/home/podman/log
/home/podman/root
/home/podman/run
/home/podman/secrets
/home/podman/storage
/home/podman/tmp
/home/podman/volumes

When trying to install some images (redis:alpine, redis:bookworm, possibly also headscale and headscale-ui, not sure), they usually fail with the following messages:

podman@<SERVERNAME>:~$ podman pull redis:alpine
✔ docker.io/library/redis:alpine
Trying to pull docker.io/library/redis:alpine...
Getting image source signatures
Copying blob 2afe905a8615 skipped: already exists  
Copying blob 4abcf2066143 skipped: already exists  
Copying blob 128c39a261ff skipped: already exists  
Copying blob f27432e97d04 skipped: already exists  
Copying blob 33486cc813b5 skipped: already exists  
Copying blob d29554ca490b skipped: already exists  
Copying blob 4f4fb700ef54 done   | 
Copying blob 5cb59ee00f00 done   | 
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": creating read-only layer with ID "1234f0cb2a613f6c85750af636ba97719670085a675695486429aef9c7530373": Stat /home/podman/storage/overlay/f5e4e1e17cc9ce821cce6f8fea6123439c6c36904d180a84075cffc3dc85483c/diff: no such file or directory

Debug level log: https://pastebin.com/F65rwZuU

I also tried podman system reset which mostly failed to delete the storage folder:

podman@<SERVERNAME>:~/compose/authentik$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/podman/storage"
        - the runRoot directory: "/run/user/1001"
Are you sure you want to continue? [y/N] Y

ERRO[0175] unlinkat /home/podman/storage/overlay/e8378ab0015b2be72773b263dc7148dcc79b2cd263832b3e5c4929d04034f641/merged: device or resource busy 
 A "/home/podman/.config/containers/storage.conf" config file exists.
Remove this file if you did not modify the configuration.
ERRO[0175] failed to remove runtime root dir /run/user/1001, since it is the same as XDG_RUNTIME_DIR 

Steps to reproduce the issue

Steps to reproduce the issue

1. Install podman on ext4 file system with the directory structure listed above
   a. This can be done mostly automatically using my helper script: https://github.com/luckylinux/podman-tools
   b. ./setup_podman_debian.sh "podman" "dir" "/home/podman"
2. Run podman pull redis:alpine

Describe the results you received

podman@<SERVERNAME>:~$ podman pull redis:alpine
✔ docker.io/library/redis:alpine
Trying to pull docker.io/library/redis:alpine...
Getting image source signatures
Copying blob 2afe905a8615 skipped: already exists  
Copying blob 4abcf2066143 skipped: already exists  
Copying blob 128c39a261ff skipped: already exists  
Copying blob f27432e97d04 skipped: already exists  
Copying blob 33486cc813b5 skipped: already exists  
Copying blob d29554ca490b skipped: already exists  
Copying blob 4f4fb700ef54 done   | 
Copying blob 5cb59ee00f00 done   | 
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": creating read-only layer with ID "1234f0cb2a613f6c85750af636ba97719670085a675695486429aef9c7530373": Stat /home/podman/storage/overlay/f5e4e1e17cc9ce821cce6f8fea6123439c6c36904d180a84075cffc3dc85483c/diff: no such file or directory

Debug level log: https://pastebin.com/F65rwZuU

Describe the results you expected

Podman pulling redis:alpine image (and others) normally.

Putting the storage folder graphRoot inside e.g. a subfolder within the user home directory works correctly apparently:

• /home/podman/containers/storage
• /home/podman/.local/share/containers/storag

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 72.8
    systemPercent: 17.26
    userPercent: 9.94
  cpus: 1
  databaseBackend: sqlite
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  freeLocks: 2048
  hostname: ra
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.1.0-18-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 1767870464
  memTotal: 2012446720
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-3_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20230309.7c7625d-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 1023406080
  swapTotal: 1023406080
  uptime: 0h 0m 19.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.13-1_amd64
      Version: |-
        fusermount3 version: 3.14.0
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /home/podman/storage
  graphRootAllocated: 18969468928
  graphRootUsed: 4682727424
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /home/podman/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1001
  transientStore: false
  volumePath: /home/podman/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.21.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

VPS on KVM AMD64.

Debian Bookwork 12 with Podman 4.9.3 pinned from Debian Testing/Trixie.

Additional information

I quickly tested in my local KVM (Proxmox VE) ZFS-based storage for podman: podman pull redis:alpine works correctly.

[luckylinux]

And now, when I tried to replicate the same directory structure on my ZFS-based installations, I encounter the issue when the folders are mounted to e.g. /home/podman/containers/{compose,config,storage,images,volumes,...}.

podman@Rock5B-02:~$ podman pull redis:alpine
✔ docker.io/library/redis:alpine
Trying to pull docker.io/library/redis:alpine...
Getting image source signatures
Copying blob bca4290a9639 skipped: already exists  
Copying blob ac791973e295 skipped: already exists  
Copying blob 2bf8baaf8aab skipped: already exists  
Copying blob dbbc6ec9f2f5 skipped: already exists  
Copying blob e214c39e91b1 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 2768c9034bfe done   | 
Error: copying system image from manifest list: trying to reuse blob sha256:e17209f7d6fed36b5abfc43936503a79fc8acd176364e78a25825d521db008c7 at destination: reading layer "a045e7de0e0a748501984bba6610ba2efc026d6c38e8cadbe27ac12f553b4931" for blob "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": 1 error occurred:
	* creating file-getter: readlink /home/podman/containers/storage/overlay/a045e7de0e0a748501984bba6610ba2efc026d6c38e8cadbe27ac12f553b4931/diff: no such file or directory

What the hell ????

Update: And of course on another server the error does NOT occur with ZFS and the new Folder Structure.

[luckylinux]

And for headscale-ui yet another error:

podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.9.3
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=headscale', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
recreating: ...
** excluding:  set()
podman stop -t 10 headscale-ui
Error: no container with name or ID "headscale-ui" found: no such container
exit code: 125
podman stop -t 10 headscale
headscale
exit code: 0
podman rm headscale-ui
Error: no container with ID or name "headscale-ui" found: no such container
exit code: 1
podman rm headscale
headscale
exit code: 0
recreating: done


['podman', 'network', 'exists', 'traefik']
podman run --name=headscale -d --label traefik.enable=true --label traefik.http.routers.headscale-rtr.rule=PathPrefix(`/`) && Host(`headscale.MYDOMAIN.TLD.TLD`) --label traefik.http.services.headscale-svc.loadbalancer.server.port=8080 --label io.podman.compose.config-hash=66e38f57d69b8f9dc2b542d2f4b011a84313ba4d23cd3f7900da773273fcd581 --label io.podman.compose.project=headscale --label io.podman.compose.version=1.0.6 --label PODMAN_SYSTEMD_UNIT=podman-compose@headscale.service --label com.docker.compose.project=headscale --label com.docker.compose.project.working_dir=/home/podman/containers/compose/headscale --label com.docker.compose.project.config_files=compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=headscale -v /home/podman/containers/config/headscale:/etc/headscale -v /home/podman/containers/data/headscale:/var/lib/headscale -v /home/podman/containers/run/headscale:/var/run/headscale --net traefik --network-alias headscale --hostname headscale.MYDOMAIN.TLD.TLD --pull always --restart unless-stopped headscale/headscale:latest-alpine headscale serve
Trying to pull docker.io/headscale/headscale:latest-alpine...
Getting image source signatures
Copying blob 3e1c99d88867 skipped: already exists  
Copying blob ca7dd9ec2225 skipped: already exists  
Copying config ff699c4c37 done   | 
Writing manifest to image destination
1a5a5ec9637b5343729d34cf5783983a1d33c1264a6013e616c8f0c9e0e6ff13
exit code: 0
['podman', 'network', 'exists', 'traefik']
podman run --name=headscale-ui -d --requires=headscale --label traefik.enable=true --label traefik.http.routers.headscale-ui-rtr.rule=PathPrefix(`/web`) && Host(`headscale.MYDOMAIN.TLD.TLD`) --label traefik.http.routers.headscale-rtr.middlewares=auth-headscale-ui --label traefik.http.services.headscale-ui-svc.loadbalancer.server.port=80 --label traefik.http.middlewares.auth-headscale-ui.basicauth.usersfile=/config/users --label io.podman.compose.config-hash=66e38f57d69b8f9dc2b542d2f4b011a84313ba4d23cd3f7900da773273fcd581 --label io.podman.compose.project=headscale --label io.podman.compose.version=1.0.6 --label PODMAN_SYSTEMD_UNIT=podman-compose@headscale.service --label com.docker.compose.project=headscale --label com.docker.compose.project.working_dir=/home/podman/containers/compose/headscale --label com.docker.compose.project.config_files=compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=headscale-ui --net traefik --network-alias headscale-ui --hostname headscale.MYDOMAIN.TLD.TLD --pull always --restart unless-stopped ghcr.io/gurucomputing/headscale-ui:latest
Trying to pull ghcr.io/gurucomputing/headscale-ui:latest...
Getting image source signatures
Copying blob 4abcf2066143 skipped: already exists  
Copying blob 7144c1805df2 skipped: already exists  
Copying blob 855024e2e210 skipped: already exists  
Copying blob 9ebf39365dcc skipped: already exists  
Copying blob 1b11e799bcea done   | 
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob f1e553f18f88 done   | 
Copying blob e770faca4e36 done   | 
Copying blob d89adfd8f9bd done   | 
Copying blob a0747d8072e8 done   | 
Copying blob 88402d33207c done   | 
Copying blob 119faa53dd31 done   | 
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob c84b632129c2 done   | 
Error: copying system image from manifest list: trying to reuse blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 at destination: reading layer "1234f0cb2a613f6c85750af636ba97719670085a675695486429aef9c7530373" for blob "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": 1 error occurred:
	* creating file-getter: readlink /home/podman/containers/images/overlay/link/home/podman/containers/storage/overlay/f5e4e1e17cc9ce821cce6f8fea6123439c6c36904d180a84075cffc3dc85483c/diff: no such file or directory


exit code: 125
podman start headscale-ui
Error: no container with name or ID "headscale-ui" found: no such container
exit code: 125
****

This was tested on another VPS and the issue could be repeated.

However, as soon as I remove the redis image, I can install headscale-ui.

So I can install either headscale-ui, or redis, NOT both !

[luckylinux]

Renaming issue since it seems more general that home root folder data storage.

Old title: Home folder ~/storage folder for graphRoot causes issues with some containers (e.g. redis:alpine)
New title: Error: copying system image from manifest list: trying to reuse blob */diff: no such file or directory

[rhatdan]

@giuseppe PTAL

[giuseppe]

I've tried your script on Debian and I encounter a lot of errors, the last one being:

./setup_tools_autoupdate_service.sh: line 23: syntax error in conditional expression

and it stops the execution.

Can you simplify the reproducer to not require such a complex configuration?

Is it enough to override some of the paths used by Podman?

Can you reproduce it on Fedora?

[luckylinux]

I've tried your script on Debian and I encounter a lot of errors, the last one being:

./setup_tools_autoupdate_service.sh: line 23: syntax error in conditional expression

I'll have to check what went wrong there. I'm using this script myself to remember what I did / what to do as there were/are several steps required in order to get Podman Rootless working. It's NOT (or was at least) plug & play.

Can you simplify the reproducer to not require such a complex configuration?
Well again, this is not required per-se. It was just what I used. Minus the syntax error.

Is it enough to override some of the paths used by Podman?

The latest findings seem to indicate that it's not necessarily caused by the folder location. /home/podman/storage works in some cases, in some cases not. /home/podman/containers/storage works in some cases, in other cases not.

But yeah ... it might be enough to change the storage / graphRoot path, as that seems where the problem occurs with overlay and overlay-images.

Can you reproduce it on Fedora?

I don't know, I don't use Fedora (and Fedora IIRC causes more issues related to SELinux being enabled by default, which I have no experience since I last used Fedora maybe 15 years ago ...)

[luckylinux]

OK there was a syntax error indeed (missing 1 "]" in "if" statement initiated by "[[") in ./setup_tools_autoupdate_service.sh: line 23.

I also updated setup_podman_debian.sh to invoke some functions AFTER the user name is defined based on command line arguments.

Fixed in the commit e8c169e I just pushed now.

[luckylinux]

I just re-ran podman-compose up -d for a headscale & headscale-ui container.

podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.9.3
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=zerotier', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
['podman', 'network', 'exists', 'zerotier_default']
podman run --name=zerotier-controller -d --label io.podman.compose.config-hash=64cb0833af58c2859dbfbc305d17bf09f42f3487a9780b73e8ea1ec3a50de667 --label io.podman.compose.project=zerotier --label io.podman.compose.version=1.0.6 --label PODMAN_SYSTEMD_UNIT=podman-compose@zerotier.service --label com.docker.compose.project=zerotier --label com.docker.compose.project.working_dir=/home/podman/containers/compose/zerotier --label com.docker.compose.project.config_files=compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=zerotier -e ZT_OVERRIDE_LOCAL_CONF=true -e ZT_ALLOW_MANAGEMENT_FROM=0.0.0.0/0 -v /home/podman/containers/data/zerotier-controller:/var/lib/zerotier-one --net zerotier_default --network-alias zerotier --expose 9993/tcp -p 9993:9993/udp --restart unless-stopped zyclonite/zerotier:latest
Error: creating container storage: the container name "zerotier-controller" is already in use by 59c03ef741df6ed4f420a0ce2a5fe107fb393b852270ca13f8c36b995b46c454. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so.
exit code: 125
podman start zerotier-controller
exit code: 0
['podman', 'network', 'exists', 'zerotier_default']
podman run --name=zerotier-ui -d --label traefik.enable=true --label traefik.http.routers.zerotier-ui-rtr.rule=PathPrefix(`/web`) && Host(`zerotier.MYDOMAIN.TLD`) --label traefik.http.routers.zerotier-ui-rtr.middlewares=zerotier-ui --label traefik.http.services.zerotier-ui-svc.loadbalancer.server.port=4000 --label io.podman.compose.config-hash=64cb0833af58c2859dbfbc305d17bf09f42f3487a9780b73e8ea1ec3a50de667 --label io.podman.compose.project=zerotier --label io.podman.compose.version=1.0.6 --label PODMAN_SYSTEMD_UNIT=podman-compose@zerotier.service --label com.docker.compose.project=zerotier --label com.docker.compose.project.working_dir=/home/podman/containers/compose/zerotier --label com.docker.compose.project.config_files=compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=zero-ui -e ZU_CONTROLLER_ENDPOINT=http://zerotier-controller:9993/ -e ZU_SECURE_HEADERS=true -e ZU_DEFAULT_USERNAME=admin -e ZU_DEFAULT_PASSWORD=zero-ui -v /home/podman/containers/data/zerotier-controller:/var/lib/zerotier-one -v /home/podman/containers/data/zerotier-ui:/app/backend/data --net zerotier_default --network-alias zero-ui --expose 4000 --restart unless-stopped dec0dos/zero-ui:latest
✔ docker.io/dec0dos/zero-ui:latest
Trying to pull docker.io/dec0dos/zero-ui:latest...
Getting image source signatures
Copying blob 6c7c9abe4a1e skipped: already exists  
Copying blob 96526aa774ef skipped: already exists  
Copying blob 824de1d006d4 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 76c34934b331 skipped: already exists  
Copying blob fdef87f136ff skipped: already exists  
Copying blob 6d9f518b8a5b skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 078acbcb3c68 done   | 
Copying blob 133eb9bb812b done   | 
Copying blob ea0d0d2be7d6 done   | 
Copying blob cab857d07113 done   | 
Copying blob e154d4509c74 done   | 
Error: copying system image from manifest list: trying to reuse blob sha256:4ee50a7ce84e2979ecdd656fd0d5ccc6a1e1ab0f9377120b30f50a9395f71f93 at destination: reading layer "fc6cc8571524708ce6078f609eb407f5ddd20be0ea58ef7fd96f3906f3566e66" for blob "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": 1 error occurred:
	* creating file-getter: readlink /home/podman/containers/images/overlay/link/home/podman/containers/storage/overlay/2c2f56a504856860ddea25ec3c23af6e1e3a6a7d92ac8d08e2d7d9e09c579e5c/diff: no such file or directory


exit code: 125
podman start zerotier-ui
Error: no container with name or ID "zerotier-ui" found: no such container
exit code: 125

Seems the path is quite long ?

/home/podman/containers/images/overlay/link/home/podman/containers/storage/overlay/2c2f56a504856860ddea25ec3c23af6e1e3a6a7d92ac8d08e2d7d9e09c579e5c/diff -> 152 characters

This part seems normal: /home/podman/containers/images/

This may make the path too long ???
overlay/link/home/podman/containers/storage/overlay/2c2f56a504856860ddea25ec3c23af6e1e3a6a7d92ac8d08e2d7d9e09c579e5c/diff

Not sure what the issue is.

podman@ServerB:~/containers/compose/zerotier$ podman run --name=zerotier-ui --log-level=debug dec0dos/zero-ui:latest
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman run --name=zerotier-ui --log-level=debug dec0dos/zero-ui:latest) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/podman/containers/storage 
DEBU[0000] Using run root /run/user/1001                
DEBU[0000] Using static dir /home/podman/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1001/libpod/tmp      
DEBU[0000] Using volume path /home/podman/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] overlay: imagestore=/home/podman/containers/storage 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 4              
DEBU[0000] Successfully loaded network traefik: &{traefik 8781be2fadcac3609b23efa839de1a015df56c8825cc04da26c4e8260b70670f bridge podman1 2024-02-25 15:28:04.904726588 +0100 CET [{{{10.89.0.0 ffffff00}} 10.89.0.1 <nil>}] [] false false true [] map[] map[] map[driver:host-local]} 
DEBU[0000] Successfully loaded network zerotier_default: &{zerotier_default a6e1e743bed72fe4279393729b8cc7a50296ef32ff16a3cefdfdc654e7d66b3d bridge podman2 2024-02-29 17:08:24.419599995 +0100 CET [{{{10.89.1.0 ffffff00}} 10.89.1.1 <nil>}] [] false false true [] map[com.docker.compose.project:zerotier io.podman.compose.project:zerotier] map[] map[driver:host-local]} 
DEBU[0000] Successfully loaded 3 networks               
DEBU[0000] Pulling image dec0dos/zero-ui:latest (policy: missing) 
DEBU[0000] Looking up image "dec0dos/zero-ui:latest" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/home/podman/.config/containers/registries.conf" 
DEBU[0000] Trying "localhost/dec0dos/zero-ui:latest" ... 
DEBU[0000] reference "[overlay@/home/podman/containers/storage+/run/user/1001:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,metacopy=on]localhost/dec0dos/zero-ui:latest" does not resolve to an image ID 
DEBU[0000] Trying "registry.fedoraproject.org/dec0dos/zero-ui:latest" ... 
DEBU[0000] reference "[overlay@/home/podman/containers/storage+/run/user/1001:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,metacopy=on]registry.fedoraproject.org/dec0dos/zero-ui:latest" does not resolve to an image ID 
DEBU[0000] Trying "registry.access.redhat.com/dec0dos/zero-ui:latest" ... 
DEBU[0000] reference "[overlay@/home/podman/containers/storage+/run/user/1001:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,metacopy=on]registry.access.redhat.com/dec0dos/zero-ui:latest" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/dec0dos/zero-ui:latest" ... 
DEBU[0000] reference "[overlay@/home/podman/containers/storage+/run/user/1001:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,metacopy=on]docker.io/dec0dos/zero-ui:latest" does not resolve to an image ID 
DEBU[0000] Trying "quay.io/dec0dos/zero-ui:latest" ...  
DEBU[0000] reference "[overlay@/home/podman/containers/storage+/run/user/1001:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,metacopy=on]quay.io/dec0dos/zero-ui:latest" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/dec0dos/zero-ui:latest" ... 
DEBU[0000] reference "[overlay@/home/podman/containers/storage+/run/user/1001:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,metacopy=on]docker.io/dec0dos/zero-ui:latest" does not resolve to an image ID 
DEBU[0000] Trying "dec0dos/zero-ui:latest" ...          
✔ docker.io/dec0dos/zero-ui:latest
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0001] Attempting to pull candidate docker.io/dec0dos/zero-ui:latest for dec0dos/zero-ui:latest 
DEBU[0001] parsed reference into "[overlay@/home/podman/containers/storage+/run/user/1001:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,metacopy=on]docker.io/dec0dos/zero-ui:latest" 
Trying to pull docker.io/dec0dos/zero-ui:latest...
DEBU[0001] Copying source image //dec0dos/zero-ui:latest to destination image [overlay@/home/podman/containers/storage+/run/user/1001:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,metacopy=on]docker.io/dec0dos/zero-ui:latest 
DEBU[0001] Using registries.d directory /etc/containers/registries.d 
DEBU[0001] Trying to access "docker.io/dec0dos/zero-ui:latest" 
DEBU[0001] No credentials matching docker.io/dec0dos/zero-ui found in /run/user/1001/containers/auth.json 
DEBU[0001] No credentials matching docker.io/dec0dos/zero-ui found in /home/podman/.config/containers/auth.json 
DEBU[0001] No credentials matching docker.io/dec0dos/zero-ui found in /home/podman/.docker/config.json 
DEBU[0001] No credentials matching docker.io/dec0dos/zero-ui found in /home/podman/.dockercfg 
DEBU[0001] No credentials for docker.io/dec0dos/zero-ui found 
DEBU[0001]  No signature storage configuration found for docker.io/dec0dos/zero-ui:latest, using built-in default file:///home/podman/.local/share/containers/sigstore 
DEBU[0001] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0001] GET https://registry-1.docker.io/v2/         
DEBU[0002] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0002] GET https://auth.docker.io/token?scope=repository%3Adec0dos%2Fzero-ui%3Apull&service=registry.docker.io 
DEBU[0002] GET https://registry-1.docker.io/v2/dec0dos/zero-ui/manifests/latest 
DEBU[0002] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0002] Using SQLite blob info cache at /home/podman/.local/share/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0002] Source is a manifest list; copying (only) instance sha256:d2a5fae1daff1dd9861623967fa7221dccf4d376cb23596c6fd469f1d8cc2c2a for current system 
DEBU[0002] GET https://registry-1.docker.io/v2/dec0dos/zero-ui/manifests/sha256:d2a5fae1daff1dd9861623967fa7221dccf4d376cb23596c6fd469f1d8cc2c2a 
DEBU[0003] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0003] IsRunningImageAllowed for image docker:docker.io/dec0dos/zero-ui:latest 
DEBU[0003]  Using default policy section                
DEBU[0003]  Requirement 0: allowed                      
DEBU[0003] Overall: allowed                             
DEBU[0003] Downloading /v2/dec0dos/zero-ui/blobs/sha256:aec81cb63f9cd23c0ae32b05b09fe4450da979ef9b9093b4894e245cee57a968 
DEBU[0003] GET https://registry-1.docker.io/v2/dec0dos/zero-ui/blobs/sha256:aec81cb63f9cd23c0ae32b05b09fe4450da979ef9b9093b4894e245cee57a968 
Getting image source signatures
DEBU[0003] Reading /home/podman/.local/share/containers/sigstore/dec0dos/zero-ui@sha256=d2a5fae1daff1dd9861623967fa7221dccf4d376cb23596c6fd469f1d8cc2c2a/signature-1 
DEBU[0003] Not looking for sigstore attachments: disabled by configuration 
DEBU[0003] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0003] ... will first try using the original manifest unmodified 
DEBU[0003] Checking if we can reuse blob sha256:6c7c9abe4a1e230d5cd151926793ca21eff1eba812bb9ad02b09db0aeebbc287: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Skipping blob sha256:6c7c9abe4a1e230d5cd151926793ca21eff1eba812bb9ad02b09db0aeebbc287 (already present): 
DEBU[0003] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Skipping blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa (already present): 
DEBU[0003] Checking if we can reuse blob sha256:824de1d006d492e037a03312c272427b62e171607bc6fb0e7db991b2eda190b7: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Skipping blob sha256:824de1d006d492e037a03312c272427b62e171607bc6fb0e7db991b2eda190b7 (already present): 
DEBU[0003] Checking if we can reuse blob sha256:76c34934b3319c116bffd4685ee2bcb032298ccce64b89b7081ec18ffdbd6175: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Skipping blob sha256:76c34934b3319c116bffd4685ee2bcb032298ccce64b89b7081ec18ffdbd6175 (already present): 
DEBU[0003] Checking if we can reuse blob sha256:fdef87f136ff3e0ba0d622cf3a573aebe6891d240d7dd55799ac0fd30a706db4: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Skipping blob sha256:fdef87f136ff3e0ba0d622cf3a573aebe6891d240d7dd55799ac0fd30a706db4 (already present): 
DEBU[0003] Checking if we can reuse blob sha256:6d9f518b8a5bd4cefa3974341d70e9328a5aff000a50d2513ce1d2b30bbb2469: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Skipping blob sha256:6d9f518b8a5bd4cefa3974341d70e9328a5aff000a50d2513ce1d2b30bbb2469 (already present): 
DEBU[0003] Checking if we can reuse blob sha256:4ee50a7ce84e2979ecdd656fd0d5ccc6a1e1ab0f9377120b30f50a9395f71f93: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Skipping blob sha256:4ee50a7ce84e2979ecdd656fd0d5ccc6a1e1ab0f9377120b30f50a9395f71f93 (already present): 
DEBU[0003] Checking if we can reuse blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Checking if we can reuse blob sha256:078acbcb3c68949ae94e09ae28130b426354f0484366c99911b2cf4f1d6c82fd: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Checking if we can reuse blob sha256:133eb9bb812b922c25b073bb272fffd6fa6c4bdc4a8e4a13ed58cbe5d83944ea: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Checking if we can reuse blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Skipping blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 (already present): 
DEBU[0003] Checking if we can reuse blob sha256:ea0d0d2be7d62c74e6b225f2ac83afd943d8c1c9a6cf41fe9aa342bc6d8e0012: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Checking if we can reuse blob sha256:cab857d07113e4ce79f705e565b9dab2b92237a7bf5506329db455a47b4f86ab: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Checking if we can reuse blob sha256:e154d4509c7422731e7381b0b8983b9392642f02b9f866e7ec19f2e9bb92ec5f: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Failed to retrieve partial blob: convert_images not configured 
DEBU[0003] Failed to retrieve partial blob: convert_images not configured 
Copying blob 6c7c9abe4a1e skipped: already exists  
Copying blob 96526aa774ef skipped: already exists  
Copying blob 824de1d006d4 skipped: already exists  
Copying blob 76c34934b331 skipped: already exists  
Copying blob fdef87f136ff skipped: already exists  
Copying blob 6d9f518b8a5b skipped: already exists  
Copying blob 4ee50a7ce84e skipped: already exists  
Copying blob 078acbcb3c68 [--------------------------------------] 0.0b / 102.9KiB (skipped: 0.0b = 0.00%)
Copying blob 133eb9bb812b [--------------------------------------] 0.0b / 959.6KiB (skipped: 0.0b = 0.00%)
Copying blob 4f4fb700ef54 skipped: already exists  
DEBU[0003] Failed to retrieve partial blob: convert_images not configured 
DEBU[0003] Failed to retrieve partial blob: convert_images not configured 
DEBU[0003] Failed to retrieve partial blob: convert_images not configured 
Copying blob 6c7c9abe4a1e skipped: already exists  
Copying blob 96526aa774ef skipped: already exists  
Copying blob 824de1d006d4 skipped: already exists  
Copying blob 76c34934b331 skipped: already exists  
Copying blob fdef87f136ff skipped: already exists  
Copying blob 6d9f518b8a5b skipped: already exists  
Copying blob 6c7c9abe4a1e skipped: already exists  
Copying blob 96526aa774ef skipped: already exists  
Copying blob 6c7c9abe4a1e skipped: already exists  
Copying blob 96526aa774ef skipped: already exists  
Copying blob 6c7c9abe4a1e skipped: already exists  
Copying blob 96526aa774ef skipped: already exists  
Copying blob 824de1d006d4 skipped: already exists  
Copying blob 76c34934b331 skipped: already exists  
Copying blob fdef87f136ff skipped: already exists  
Copying blob 6d9f518b8a5b skipped: already exists  
Copying blob 4ee50a7ce84e skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 078acbcb3c68 done   | 
Copying blob 133eb9bb812b done   | 
Copying blob ea0d0d2be7d6 done   | 
Copying blob cab857d07113 done   | 
Copying blob e154d4509c74 done   | 
DEBU[0005] Error pulling candidate docker.io/dec0dos/zero-ui:latest: copying system image from manifest list: trying to reuse blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 at destination: reading layer "fc6cc8571524708ce6078f609eb407f5ddd20be0ea58ef7fd96f3906f3566e66" for blob "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": 1 error occurred:
	* creating file-getter: readlink /home/podman/containers/images/overlay/link/home/podman/containers/storage/overlay/2c2f56a504856860ddea25ec3c23af6e1e3a6a7d92ac8d08e2d7d9e09c579e5c/diff: no such file or directory
 
Error: copying system image from manifest list: trying to reuse blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 at destination: reading layer "fc6cc8571524708ce6078f609eb407f5ddd20be0ea58ef7fd96f3906f3566e66" for blob "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": 1 error occurred:
	* creating file-getter: readlink /home/podman/containers/images/overlay/link/home/podman/containers/storage/overlay/2c2f56a504856860ddea25ec3c23af6e1e3a6a7d92ac8d08e2d7d9e09c579e5c/diff: no such file or directory


DEBU[0005] Shutting down engines 

[luckylinux]

OK I acknowledge there were several errors, as you @giuseppe reported.

Hopefully I fixed most of them now in my latest commits, but I ended up in a new one that I never saw before.

Keep in mind this is on a Raspberry Pi 2 (armv7l) so it might also be related to ARM 32 bit instruction set (even more limited than armhf - yeah confusing Raspberry Pi logic in the architecture naming).

root@RaspberryPI-Flashed:/tools_local/podman-tools # ./setup_podman_debian.sh "podman" "dir" "/home/podman/containers"
useradd: user 'podman' already exists
passwd: password changed.
New password: 
Retype new password: 
passwd: password updated successfully
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
sudo is already the newest version (1.9.13p3-1+deb12u1).
aptitude is already the newest version (0.8.13-5).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
podman is already the newest version (4.3.1+ds1-8).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3 is already the newest version (3.11.2-1).
python3-pip is already the newest version (23.0.1+dfsg-1+rpt1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Please add <systemd.unified_cgroup_hierarchy=1> to /etc/default/kernel-cmdline or /etc/default/grub
Press ENTER once ready
--2024-03-02 20:11:35--  https://src.fedoraproject.org/rpms/containers-common/raw/main/f/storage.conf
Resolving src.fedoraproject.org (src.fedoraproject.org)... 38.145.60.20, 38.145.60.21
Connecting to src.fedoraproject.org (src.fedoraproject.org)|38.145.60.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10899 (11K) [text/plain]
Saving to: ‘storage.conf’

storage.conf                                                100%[==========================================================================================================================================>]  10.64K  --.-KB/s    in 0.002s  

2024-03-02 20:11:36 (5.30 MB/s) - ‘storage.conf’ saved [10899/10899]

--2024-03-02 20:11:36--  https://src.fedoraproject.org/rpms/containers-common/raw/main/f/registries.conf
Resolving src.fedoraproject.org (src.fedoraproject.org)... 38.145.60.20, 38.145.60.21
Connecting to src.fedoraproject.org (src.fedoraproject.org)|38.145.60.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3895 (3.8K) [text/plain]
Saving to: ‘registries.conf’

registries.conf                                             100%[==========================================================================================================================================>]   3.80K  --.-KB/s    in 0s      

2024-03-02 20:11:37 (15.5 MB/s) - ‘registries.conf’ saved [3895/3895]

--2024-03-02 20:11:37--  https://src.fedoraproject.org/rpms/containers-common/raw/main/f/default-policy.json
Resolving src.fedoraproject.org (src.fedoraproject.org)... 38.145.60.21, 38.145.60.20
Connecting to src.fedoraproject.org (src.fedoraproject.org)|38.145.60.21|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 256 [application/json]
Saving to: ‘default-policy.json’

default-policy.json                                         100%[==========================================================================================================================================>]     256  --.-KB/s    in 0s      

2024-03-02 20:11:37 (1.05 MB/s) - ‘default-policy.json’ saved [256/256]

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
uidmap is already the newest version (1:4.13+dfsg1-1).
fuse-overlayfs is already the newest version (1.10-1).
slirp4netns is already the newest version (1.2.0-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Created symlink /home/podman/.config/systemd/user/sockets.target.wants/podman.socket → /home/podman/.config/systemd/user/podman.socket.
Created symlink /home/podman/.config/systemd/user/default.target.wants/podman.service → /home/podman/.config/systemd/user/podman.service.
Created symlink /home/podman/.config/systemd/user/default.target.wants/podman-restart.service → /home/podman/.config/systemd/user/podman-restart.service.
Job for podman-restart.service failed because the control process exited with error code.
See "systemctl --user status podman-restart.service" and "journalctl --user -xeu podman-restart.service" for details.
Created symlink /home/podman/.config/systemd/user/default.target.wants/podman-auto-update.service → /home/podman/.config/systemd/user/podman-auto-update.service.
Job for podman-auto-update.service failed because the control process exited with error code.
See "systemctl --user status podman-auto-update.service" and "journalctl --user -xeu podman-auto-update.service" for details.
● podman.socket - Podman API Socket
     Loaded: loaded (/home/podman/.config/systemd/user/podman.socket; enabled; preset: enabled)
     Active: active (listening) since Sat 2024-03-02 20:12:07 CET; 8s ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1001/podman/podman.sock (Stream)
     CGroup: /user.slice/user-1001.slice/user@1001.service/app.slice/podman.socket

Mar 02 20:12:07 RaspberryPI-Flashed systemd[2472]: Listening on podman.socket - Podman API Socket.

× podman.service - Podman API Service
     Loaded: loaded (/home/podman/.config/systemd/user/podman.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2024-03-02 20:12:11 CET; 3s ago
   Duration: 2.303s
TriggeredBy: ● podman.socket
       Docs: man:podman-system-service(1)
   Main PID: 6985 (code=exited, status=125)
        CPU: 550ms

Mar 02 20:12:09 RaspberryPI-Flashed systemd[2472]: Starting podman.service - Podman API Service...
Mar 02 20:12:09 RaspberryPI-Flashed systemd[2472]: Started podman.service - Podman API Service.
Mar 02 20:12:09 RaspberryPI-Flashed podman[6985]: time="2024-03-02T20:12:09+01:00" level=info msg="/usr/bin/podman filtering at log level info"
Mar 02 20:12:09 RaspberryPI-Flashed podman[6985]: Error: failed to get new shm lock manager: failed to create 2048 locks in /libpod_rootless_lock_1001: permission denied
Mar 02 20:12:11 RaspberryPI-Flashed systemd[2472]: podman.service: Main process exited, code=exited, status=125/n/a
Mar 02 20:12:11 RaspberryPI-Flashed systemd[2472]: podman.service: Failed with result 'exit-code'.

× podman-restart.service - Podman Start All Containers With Restart Policy Set To Always
     Loaded: loaded (/home/podman/.config/systemd/user/podman-restart.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2024-03-02 20:12:12 CET; 3s ago
       Docs: man:podman-start(1)
   Main PID: 7038 (code=exited, status=125)
        CPU: 329ms

Mar 02 20:12:12 RaspberryPI-Flashed systemd[2472]: Starting podman-restart.service - Podman Start All Containers With Restart Policy Set To Always...
Mar 02 20:12:12 RaspberryPI-Flashed podman[7038]: time="2024-03-02T20:12:12+01:00" level=info msg="/usr/bin/podman filtering at log level info"
Mar 02 20:12:12 RaspberryPI-Flashed podman[7038]: Error: failed to get new shm lock manager: failed to create 2048 locks in /libpod_rootless_lock_1001: permission denied
Mar 02 20:12:12 RaspberryPI-Flashed systemd[2472]: podman-restart.service: Main process exited, code=exited, status=125/n/a
Mar 02 20:12:12 RaspberryPI-Flashed systemd[2472]: podman-restart.service: Failed with result 'exit-code'.
Mar 02 20:12:12 RaspberryPI-Flashed systemd[2472]: Failed to start podman-restart.service - Podman Start All Containers With Restart Policy Set To Always.

× podman-auto-update.service - Podman auto-update service
     Loaded: loaded (/home/podman/.config/systemd/user/podman-auto-update.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2024-03-02 20:12:15 CET; 259ms ago
       Docs: man:podman-auto-update(1)
    Process: 7083 ExecStart=/usr/bin/podman auto-update (code=exited, status=125)
   Main PID: 7083 (code=exited, status=125)
        CPU: 327ms

Mar 02 20:12:14 RaspberryPI-Flashed systemd[2472]: Starting podman-auto-update.service - Podman auto-update service...
Mar 02 20:12:15 RaspberryPI-Flashed podman[7083]: Error: failed to get new shm lock manager: failed to create 2048 locks in /libpod_rootless_lock_1001: permission denied
Mar 02 20:12:15 RaspberryPI-Flashed systemd[2472]: podman-auto-update.service: Main process exited, code=exited, status=125/n/a

And the key error is
Error: failed to get new shm lock manager: failed to create 2048 locks in /libpod_rootless_lock_1001: permission denied

[luckylinux]

Any update ?

[giuseppe]

who owns /dev/shm/libpod_rootless_lock_1001?

Can your user create that directory?

[luckylinux]

who owns /dev/shm/libpod_rootless_lock_1001?

Can your user create that directory?

I cannot say it is an issue during deployment. It was an error during SETUP, something that didn't occur in the past

podman@RaspberryPI-ChargerDevelopment:~ $ ls -l /dev/shm/libpod_rootless_lock_1001 
-rw------- 1 podman podman 49444 Mar 10 18:27 /dev/shm/libpod_rootless_lock_1001

It's fair for you to say that it's difficult to replicate as I'm "only" encountering it 50% of the time. So it's kinda hit-or-miss.

I fixed some bugs in the scripts (thank you for reporting that), but it also doesn't always happen.

However, I can say now that both ZFS+rbind ("zfs") and EXT4 ("dir") are affected. And not only on AMD64.