'podman system df' is slow

[Nuc1eoN]

podman system df in rootless mode hangs indefinitely for me.

Can anybody reproduce?

[rhatdan]

It works for me.
Does podman ps -a and podman images take a long time?

[Nuc1eoN]

podman images is instant for me.

Also a little update. I let podman system df run for a while and it turns out it does actually finish but the process takes ~3 minutes.

[paravz]

journalctl -l --all --no-pager -f while sudo podman system df is running shows that all (most?) image layers is being mounted during this process, takes 5 minutes on my system, output is funky to say the least (containers were created using podman 1.5 or 1.6, now running F32's podman 2.0.4):

$ time sudo podman system df -v 2>&1 | tee -a podman-2.0.4-system-df.log

Images space usage:

REPOSITORY                         TAG       IMAGE ID      CREATED    SIZE     SHARED SIZE  UNIQUE SIZE  CONTAINERS
<none>                             <none>    289289d1a15b  14 months  326.2MB  0B           326.2MB      1
localhost/test                     d0a6310   bf62eb48712c  14 months  3.771GB  0B           3.771GB      4
<none>                             <none>    e06a00c9c1d7  13 months  326.2MB  326.2MB      -3333B       1
<none>                             <none>    b49d797fed63  13 months  326.2MB  326.2MB      -1.381e+04B  1
<none>                             <none>    72979d8140b0  13 months  326.2MB  326.2MB      -2.712e+04B  1
<none>                             <none>    91c62fb55d3a  13 months  332.6MB  338.9MB      -6.368e+06B  1
<none>                             <none>    d1ac9ff6e033  13 months  1.357GB  2.381GB      -1.024e+09B  1
<none>                             <none>    0c1e23b2d31c  13 months  1.357GB  1.357GB      -3.212e+05B  1
<none>                             <none>    7433b67d788e  13 months  1.364GB  1.371GB      -6.777e+06B  1
<none>                             <none>    fba2cc484d33  13 months  1.399GB  1.435GB      -3.539e+07B  1
<none>                             <none>    81f74ebe319e  13 months  1.399GB  1.399GB      -1.124e+04B  1
<none>                             <none>    e4c8edf468a6  13 months  1.399GB  1.399GB      -4.451e+04B  1
<none>                             <none>    6d03e3f2fc06  13 months  1.399GB  1.399GB      -4.451e+04B  1
<none>                             <none>    e5c52a1880a5  13 months  326.2MB  326.2MB      -3333B       0
<none>                             <none>    2b8114581ee4  13 months  326.2MB  326.2MB      -1.381e+04B  0
<none>                             <none>    05f71c6eeb8e  13 months  326.2MB  326.2MB      -2.712e+04B  0
<none>                             <none>    d78678de3e20  13 months  332.6MB  338.9MB      -6.368e+06B  0
<none>                             <none>    490baf730067  13 months  1.357GB  2.381GB      -1.024e+09B  0
<none>                             <none>    3fa5d6820708  13 months  1.357GB  1.357GB      -3.212e+05B  0
<none>                             <none>    f0e5ee5846e0  13 months  1.364GB  1.371GB      -6.777e+06B  0
<none>                             <none>    ee798016f66b  13 months  1.399GB  1.434GB      -3.539e+07B  0
<none>                             <none>    ae3ca2acb94e  13 months  1.399GB  1.399GB      -1.124e+04B  0
<none>                             <none>    4a05cef439a9  13 months  1.399GB  1.399GB      -4.451e+04B  0
<none>                             <none>    7f93e174adbe  13 months  1.399GB  1.399GB      -4.451e+04B  0
<none>                             <none>    f30c316ff16b  13 months  3.664GB  5.928GB      -2.264e+09B  0
<none>                             <none>    604f1dbf9b24  13 months  3.664GB  3.664GB      -1.747e+04B  0
<trimmed>

Note pid 1 and systemd --user's pid in log (trimmed):

systemd[1]: var-lib-containers-storage-overlay-cdfd8542e233f64902ae0e217d3c024ed07ea879ff8a42dd17dfd5760688fdbe-merged.mount: Succeeded.
systemd[1463]: var-lib-containers-storage-overlay-cdfd8542e233f64902ae0e217d3c024ed07ea879ff8a42dd17dfd5760688fdbe-merged.mount: Succeeded.
kernel: overlayfs: lowerdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
kernel: overlayfs: lowerdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
systemd[1]: var-lib-containers-storage-overlay-cdfd8542e233f64902ae0e217d3c024ed07ea879ff8a42dd17dfd5760688fdbe-merged.mount: Succeeded.
systemd[1463]: var-lib-containers-storage-overlay-cdfd8542e233f64902ae0e217d3c024ed07ea879ff8a42dd17dfd5760688fdbe-merged.mount: Succeeded.
systemd[1463]: var-lib-containers-storage-overlay-1a3669dbba07ed4bb912fffbafcf777bfe85af54616cf1ea382e06349db1ba92-merged.mount: Succeeded.
systemd[1]: var-lib-containers-storage-overlay-1a3669dbba07ed4bb912fffbafcf777bfe85af54616cf1ea382e06349db1ba92-merged.mount: Succeeded.
systemd[1]: var-lib-containers-storage-overlay-98acca75f0b46e527d6ffaaf547e3f9b8743c70ff84daa7f0001128821158741-merged.mount: Succeeded.

Edit: also related to #7405

[rhatdan]

@vrothberg PTAL

[rhatdan]

Seems similar to the issues you took care of with images.

[vrothberg]

I'll take a look, sure. A friendly reminder to use our issue template as it's invaluable to have data about the host and its environment.

[vrothberg]

Eeek ... looking deeper, the image sizes reported by df --verbose are wrong. I'll tackle that as well.

[vrothberg]

#7604 to the rescue