[Error] When trying to create a Distrobox container on a system via rootless Podman with a user managed by systemd-homed, it fails after pulling the image.

[IPlayZed]

Describe the bug
When trying to create a Distrobox container on a system via rootless Podman with a user managed by systemd-homed, it fails after oulling the image.

To Reproduce

1. Log into a user managed by Systemd-Homed.
2. Issue distrobox create.
3. Pull is OK, but crashes after trying to set up the container.

Expected behavior
A clear and concise description of what you expected to happen.

Logs
Run the commands with --verbose and post the log here as a file upload:

❯ distrobox create --verbose
+ '[' -z '' ']'
+ '[' -z '' ']'
+ container_image=registry.fedoraproject.org/fedora-toolbox:38
+ '[' -z '' ']'
+ '[' registry.fedoraproject.org/fedora-toolbox:38 = registry.fedoraproject.org/fedora-toolbox:38 ']'
+ container_name=my-distrobox
+ '[' -z my-distrobox ']'
+ case "${container_manager}" in
+ command -v podman
+ container_manager=podman
+ command -v podman
+ '[' 1 -ne 0 ']'
+ container_manager='podman --log-level debug'
+ '[' 0 -ne 0 ']'
+ '[' -z /usr/bin/distrobox-init ']'
+ '[' -z /usr/bin/distrobox-export ']'
+ '[' 0 -ne 0 ']'
+ podman --log-level debug inspect --type container my-distrobox
+ '[' -n '' ']'
+ '[' 0 -eq 1 ']'
+ podman --log-level debug inspect --type image registry.fedoraproject.org/fedora-toolbox:38
+ '[' 0 -eq 1 ']'
+ '[' 0 -eq 1 ']'
+ printf 'Image %s not found.\n' registry.fedoraproject.org/fedora-toolbox:38
Image registry.fedoraproject.org/fedora-toolbox:38 not found.
+ printf 'Do you want to pull the image now? [Y/n]: '
Do you want to pull the image now? [Y/n]: + read -r response
Y
+ response=Y
+ case "${response}" in
+ podman --log-level debug pull registry.fedoraproject.org/fedora-toolbox:38
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level debug pull registry.fedoraproject.org/fedora-toolbox:38) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/personal/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/personal/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/60311/containers    
DEBU[0000] Using static dir /home/personal/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/60311/libpod/tmp     
DEBU[0000] Using volume path /home/personal/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: storage already configured with a mount-program 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 97             
DEBU[0000] Pulling image registry.fedoraproject.org/fedora-toolbox:38 (policy: always) 
DEBU[0000] Looking up image "registry.fedoraproject.org/fedora-toolbox:38" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "registry.fedoraproject.org/fedora-toolbox:38" ... 
DEBU[0000] reference "[overlay@/home/personal/.local/share/containers/storage+/run/user/60311/containers]registry.fedoraproject.org/fedora-toolbox:38" does not resolve to an image ID 
DEBU[0000] Trying "registry.fedoraproject.org/fedora-toolbox:38" ... 
DEBU[0000] reference "[overlay@/home/personal/.local/share/containers/storage+/run/user/60311/containers]registry.fedoraproject.org/fedora-toolbox:38" does not resolve to an image ID 
DEBU[0000] Trying "registry.fedoraproject.org/fedora-toolbox:38" ... 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf" 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate registry.fedoraproject.org/fedora-toolbox:38 for registry.fedoraproject.org/fedora-toolbox:38 
DEBU[0000] parsed reference into "[overlay@/home/personal/.local/share/containers/storage+/run/user/60311/containers]registry.fedoraproject.org/fedora-toolbox:38" 
Trying to pull registry.fedoraproject.org/fedora-toolbox:38...
DEBU[0000] Copying source image //registry.fedoraproject.org/fedora-toolbox:38 to destination image [overlay@/home/personal/.local/share/containers/storage+/run/user/60311/containers]registry.fedoraproject.org/fedora-toolbox:38 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "registry.fedoraproject.org/fedora-toolbox:38" 
DEBU[0000] No credentials matching registry.fedoraproject.org/fedora-toolbox found in /run/user/60311/containers/auth.json 
DEBU[0000] No credentials matching registry.fedoraproject.org/fedora-toolbox found in /home/personal/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.fedoraproject.org/fedora-toolbox found in /home/personal/.docker/config.json 
DEBU[0000] No credentials matching registry.fedoraproject.org/fedora-toolbox found in /home/personal/.dockercfg 
DEBU[0000] No credentials for registry.fedoraproject.org/fedora-toolbox found 
DEBU[0000]  No signature storage configuration found for registry.fedoraproject.org/fedora-toolbox:38, using built-in default file:///home/personal/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fedoraproject.org 
DEBU[0000] GET https://registry.fedoraproject.org/v2/   
DEBU[0000] Ping https://registry.fedoraproject.org/v2/ status 200 
DEBU[0000] GET https://registry.fedoraproject.org/v2/fedora-toolbox/manifests/38 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0001] Using blob info cache at /home/personal/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:5f470e939f836be0314e28fb438a58b902b8a939ca4a0bb448fe1a8a7c1e283c for current system 
DEBU[0001] GET https://registry.fedoraproject.org/v2/fedora-toolbox/manifests/sha256:5f470e939f836be0314e28fb438a58b902b8a939ca4a0bb448fe1a8a7c1e283c 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0001] IsRunningImageAllowed for image docker:registry.fedoraproject.org/fedora-toolbox:38 
DEBU[0001]  Using default policy section                
DEBU[0001]  Requirement 0: allowed                      
DEBU[0001] Overall: allowed                             
DEBU[0001] Downloading /v2/fedora-toolbox/blobs/sha256:bfe872bb72c1922b94e3cdcfb6ae8511d55e98860effa6bb7a76f1acc7a892eb 
DEBU[0001] GET https://registry.fedoraproject.org/v2/fedora-toolbox/blobs/sha256:bfe872bb72c1922b94e3cdcfb6ae8511d55e98860effa6bb7a76f1acc7a892eb 
Getting image source signatures
DEBU[0001] Reading /home/personal/.local/share/containers/sigstore/fedora-toolbox@sha256=5f470e939f836be0314e28fb438a58b902b8a939ca4a0bb448fe1a8a7c1e283c/signature-1 
DEBU[0001] Not looking for sigstore attachments: disabled by configuration 
DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0001] ... will first try using the original manifest unmodified 
DEBU[0001] Checking if we can reuse blob sha256:24da51ec3f05bb3fed8a94842047d17ddae58a6f21c30faa2e42121f1c76b8df: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0001] Checking if we can reuse blob sha256:f68bc6b0276984fbb8ef2fe288f9086dfa85c9fb2e4e439ff2f2b5034b1aad27: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0001] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0001] Downloading /v2/fedora-toolbox/blobs/sha256:24da51ec3f05bb3fed8a94842047d17ddae58a6f21c30faa2e42121f1c76b8df 
DEBU[0001] GET https://registry.fedoraproject.org/v2/fedora-toolbox/blobs/sha256:24da51ec3f05bb3fed8a94842047d17ddae58a6f21c30faa2e42121f1c76b8df 
DEBU[0001] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0001] Downloading /v2/fedora-toolbox/blobs/sha256:f68bc6b0276984fbb8ef2fe288f9086dfa85c9fb2e4e439ff2f2b5034b1aad27 
DEBU[0001] GET https://registry.fedoraproject.org/v2/fedora-toolbox/blobs/sha256:f68bc6b0276984fbb8ef2fe288f9086dfa85c9fb2e4e439ff2f2b5034b1aad27 
Copying blob 24da51ec3f05 [--------------------------------------] 0.0b / 220.1MiB (skipped: 0.0b = 0.00%)
Copying blob 24da51ec3f05 [--------------------------------------] 0.0b / 220.1MiB
Copying blob 24da51ec3f05 [--------------------------------------] 0.0b / 220.1MiB (skipped: 0.0b = 0.00%)
Copying blob 24da51ec3f05 [--------------------------------------] 1.3MiB / 220.1MiB
Copying blob 24da51ec3f05 [==================>-------------------] 110.7MiB / 220.1MiB
Copying blob 24da51ec3f05 done  
Copying blob f68bc6b02769 done  
DEBU[0009] Error pulling candidate registry.fedoraproject.org/fedora-toolbox:38: copying system image from manifest list: writing blob: adding layer with blob "sha256:f68bc6b0276984fbb8ef2fe288f9086dfa85c9fb2e4e439ff2f2b5034b1aad27": processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 0:12 for /var/spool/mail): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /var/spool/mail: invalid argument): exit status 1 
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:f68bc6b0276984fbb8ef2fe288f9086dfa85c9fb2e4e439ff2f2b5034b1aad27": processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 0:12 for /var/spool/mail): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /var/spool/mail: invalid argument): exit status 1
DEBU[0009] Shutting down engines   

Attach also the output of podman logs or docker logs, possibly with --latest flag:

❯ podman logs --latest
Error: no such container

Desktop (please complete the following information):

• Are you using podman, docker or lilipod?: Podman, in a rootless configuration.
• Which version or podman, docker or lilipod?: podman version 4.6.2
• Which version of distrobox?: distrobox version: 1.5.0.2
• Which host distribution?: Arch 6.4.15-hardened1-1-hardened
• How did you install distrobox?: Via Pacman.

Additional context
This happens only if the user is managed by Systemd-Homed.
I set up rootless Podman and as described on the Arch Wiki here the UIDs are correct for container users, but probably not the GIDs, as they are not set (hence the - I guess?):

❯ userdbctl
   NAME                           DISPOSITION        UID   GID REALNAME                         HOME                     SHELL
   root                           intrinsic            0     0 -                                /root                    /bin/bash
┌─ ↓ begin system users ↓         system               1     - First system user                -                        -
   bin                            system               1     1 -                                /                        /usr/bin/nologin
   daemon                         system               2     2 -                                /                        /usr/bin/nologin
   mail                           system               8    12 -                                /var/spool/mail          /usr/bin/nologin
   ftp                            system              14    11 -                                /srv/ftp                 /usr/bin/nologin
   rpc                            system              32    32 Rpcbind Daemon                   /var/lib/rpcbind         /usr/bin/nologin
   http                           system              33    33 -                                /srv/http                /usr/bin/nologin
   clamav                         system              64    64 Clam AntiVirus                   /                        /usr/bin/nologin
   uuidd                          system              68    68 -                                /                        /usr/bin/nologin
   dbus                           system              81    81 System Message Bus               /                        /usr/bin/nologin
   polkitd                        system             102   102 PolicyKit daemon                 /                        /usr/bin/nologin
   gdm                            system             120   120 Gnome Display Manager            /var/lib/gdm             /usr/bin/nologin
   rtkit                          system             133   133 RealtimeKit                      /proc                    /usr/bin/nologin
   usbmux                         system             140   140 usbmux user                      /                        /usr/bin/nologin
   nvidia-persistenced            system             143   143 NVIDIA Persistence Daemon        /                        /usr/bin/nologin
   cups                           system             209   209 cups helper user                 /                        /usr/bin/nologin
   fwupd                          system             951   951 Firmware update daemon           /var/lib/fwupd           /usr/bin/nologin
   systemd-journal-upload         system             953   953 systemd Journal Upload           /                        /usr/bin/nologin
   saned                          system             955   955 SANE daemon user                 /                        /usr/bin/nologin
   mysql                          system             956   956 MariaDB                          /var/lib/mysql           /usr/bin/nologin
   sddm                           system             958   958 Simple Desktop Display Manager   /var/lib/sddm            /usr/bin/nologin
   nm-openconnect                 system             960   960 NetworkManager OpenConnect       /                        /usr/bin/nologin
   openvpn                        system             961   961 OpenVPN                          /                        /usr/bin/nologin
   nm-openvpn                     system             962   962 NetworkManager OpenVPN           /                        /usr/bin/nologin
   dnsmasq                        system             963   963 dnsmasq daemon                   /                        /usr/bin/nologin
   qemu                           system             964   964 QEMU user                        /                        /usr/bin/nologin
   libvirt-qemu                   system             965   965 Libvirt QEMU user                /                        /usr/bin/nologin
   gluster                        system             966   966 GlusterFS daemons                /var/run/gluster         /usr/bin/nologin
   git                            system             968   968 git daemon user                  /                        /usr/bin/git-shell
   geoclue                        system             969   969 Geoinformation service           /var/lib/geoclue         /usr/bin/nologin
   flatpak                        system             970   970 Flatpak system helper            /                        /usr/bin/nologin
   colord                         system             971   971 Color management daemon          /var/lib/colord          /usr/bin/nologin
   brltty                         system             972   972 Braille Device Daemon            /var/lib/brltty          /usr/bin/nologin
   avahi                          system             973   973 Avahi mDNS/DNS-SD daemon         /                        /usr/bin/nologin
   tss                            system             975   975 tss user for tpm2                /                        /usr/bin/nologin
   systemd-timesync               system             976   976 systemd Time Synchronization     /                        /usr/bin/nologin
   systemd-resolve                system             977   977 systemd Resolver                 /                        /usr/bin/nologin
   systemd-journal-remote         system             978   978 systemd Journal Remote           /                        /usr/bin/nologin
   systemd-oom                    system             979   979 systemd Userspace OOM Killer     /                        /usr/bin/nologin
   systemd-network                system             980   980 systemd Network Management       /                        /usr/bin/nologin
   systemd-coredump               system             981   981 systemd Core Dumper              /                        /usr/bin/nologin
└─ ↑ end system users ↑           system             999     - Last system user                 -                        -
   developer-no-homed             regular           1000  1000 developer (not Homed managed)    /home/developer-no-homed /bin/bash
┌─ ↓ begin systemd-homed users ↓  regular          60001     - First systemd-homed user         -                        -
   gaming                         regular          60197 60197 Börcsök Balázs Róbert (gaming)   /home/gaming             /bin/bash
   admin                          regular          60282 60282 Börcsök Balázs Róbert (admin)    /home/admin              /bin/bash
   personal                       regular          60311 60311 Börcsök Balázs Róbert (personal) /home/personal           /bin/bash
└─ ↑ end systemd-homed users ↑    regular          60513     - Last systemd-homed user          -                        -
┌─ ↓ begin mapped users ↓         regular          60514     - First mapped user                -                        -
└─ ↑ end mapped users ↑           regular          60577     - Last mapped user                 -                        -
┌─ ↓ begin dynamic system users ↓ dynamic          61184     - First dynamic system user        -                        -
└─ ↑ end dynamic system users ↑   dynamic          65519     - Last dynamic system user         -                        -
   nobody                         intrinsic        65534 65534 Kernel Overflow User             /                        /usr/bin/nologin
┌─ ↓ begin container users ↓      container       524288     - First container user             -                        -
└─ ↑ end container users ↑        container   1878982656     - Last container user              -                        -

46 users listed.

[89luca89]

Hi @IPlayZed

Does this work if you simply use podman?

podman run --rm -ti --userns keep-id -v /home:/home registry.fedoraproject.org/fedora-toolbox:38 /bin/bash

[IPlayZed]

Hi @IPlayZed

Does this work if you simply use podman?

podman run --rm -ti --userns keep-id -v /home:/home registry.fedoraproject.org/fedora-toolbox:38 /bin/bash

As I was suspecting, the exact same behaviour :(.
Then it seems like this is a Podman+Homed problem.

[89luca89]

Yep that's a podman problem
So It should be reported to podman (with the podman command you tried)

Not distrobox related

[IPlayZed]

Also, this is a duplicate of #97