Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage utilisation doesn't seem to add-up... #13516

Closed
srcshelton opened this issue Mar 15, 2022 · 8 comments · Fixed by #14755
Closed

Storage utilisation doesn't seem to add-up... #13516

srcshelton opened this issue Mar 15, 2022 · 8 comments · Fixed by #14755
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@srcshelton
Copy link
Contributor

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Could anyone explain this output?

# podman system prune -f
Deleted Images
Total reclaimed space: 0B

# podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         111         19          45.22GB     38.73GB (0%)
Containers     20          0           3.806GB     3.806GB (100%)
Local Volumes  38          0           569.8MB     1.585GB (200%)

# podman volume prune -f
3081eba3e280889614b140a06368543a932110679a1e376c9ed3698ee4a51e3b
c0bf3bfa98b8b62559615fd6f8845d01a384c4ed89d71905223c585f7b09180f

# podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         111         19          45.22GB     38.73GB (0%)
Containers     20          0           3.806GB     3.806GB (100%)
Local Volumes  36          0           371.9MB     0B (0%)

… as a side-note, I've often seem podman system prune returning what appear to be feasibly high figures (… larger than the containing filesystem, in some cases) when reporting on storage savings - I always assumed this was double-counting shared layers, or similar...

The output above seems to indicate that system prune did not actually "Remove all unused pod, container, image and volume data" but only removed images and stopped containers, but then somehow we were left with a state where podman considered there to be 3 times as much reclaimable volume space as the amount of storage that existing volumes were actually consuming, and in any case reported this as 200% (which doesn't appear to indicate "200% more on top", as the container reclaimable space of 3.8GB out of 3.8GB is listed as 100%. You'd assume that for volumes with 570MB reclaimable the percentage should likewise be 100%, so 1.585GB should be about 279%… although, from the final output, the true figure looks as if it should have been 34.7%?)

Could having two containers which both inherit the same volume from a (terminated) progenitor container cause some form of double-counting error?

Also, in this case all running containers are paused - is that perhaps causing the reclaimable Container storage to be misrepresented as 100%?

Steps to reproduce the issue:

  1. podman system prune -f;

  2. Observe amount of space reported to have been cleared;

  3. Observe podman system df output.

Describe the results you received:

  • The prune operation did not reclaim all reclaimable space;

  • The sizes and percentages don't appear to be internally consistent, or map to real-world disk utilisation.

Describe the results you expected:

  • A system prune operation should surely reclaim the maximum amount of storage (handling image/container/volume dependencies as necessary)?

  • Percentages and utilisation figures should match-up, reports of storage space consumed and freed should match filesystem usage data.

Output of podman version:

Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.17.7
Git Commit:   342c8259381b63296e96ad29519bd4b9c7afbf97
Built:        Mon Mar  7 01:21:28 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: app-containers/conmon-2.1.0
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
  cpus: 8
  distribution:
    distribution: gentoo
    version: unknown
  eventLogger: file
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.16.12-gentoo
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 943378432
  memTotal: 68205772800
  networkBackend: cni
  ociRuntime:
    name: crun
    package: app-containers/crun-1.4.3
    path: /usr/bin/crun
    version: |-
      crun version 1.4.3
      commit: 61c9600d1335127eba65632731e2d72bc3f0b9e8
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.1.12
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 21388185600
  swapTotal: 25769787392
  uptime: 50h 23m 28.56s (Approximately 2.08 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: localhost:5000
  search:
  - docker.io
  - docker.pkg.github.com
  - quay.io
  - public.ecr.aws
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 20
    paused: 20
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /space/podman/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp/.private/root
  imageStore:
    number: 109
  runRoot: /var/run/podman
  volumePath: /space/podman/volumes
version:
  APIVersion: 4.0.2
  Built: 1646616088
  BuiltTime: Mon Mar  7 01:21:28 2022
  GitCommit: 342c8259381b63296e96ad29519bd4b9c7afbf97
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 15, 2022
@vrothberg
Copy link
Member

Thanks for opening the issue, @srcshelton!

It seems very odd that the volumes aren't removed with system prune. For the other issues, it would be ideal to have reproducers.

@containers/podman-maintainers FYI

@srcshelton
Copy link
Contributor Author

I'll see what I can do reproducer-wise, but it's probably going to be a tricky one to catch other than anecdotally...

@srcshelton
Copy link
Contributor Author

Just from general use, tonight's output:

$ sudo podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         189         23          86.27GB     78.68GB (0%)
Containers     24          23          4.783MB     3.472MB (0%)
Local Volumes  46          48          701MB       0B (0%)

(78GB is obviously not 0% of 86%, and 4.7MB for 24 containers seems a little low? Also, the output is showing more Active Local Volumes than Total Local Volumes...)

$ df -h /space/podman/
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/storage-space  1.6T  137G  1.5T   9% /space

$ sudo du -chxs /space/podman/*
2.3M    /space/podman/static
44G     /space/podman/storage
631M    /space/podman/tmp
1.3G    /space/podman/volumes
46G     total

$ sudo podman system prune -f
<…>
Total reclaimed space: 37.22GB

$ df -h /space/podman/
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/storage-space  1.6T  125G  1.5T   8% /space

$ sudo du -chxs /space/podman/*
2.3M    /space/podman/static
33G     /space/podman/storage
631M    /space/podman/tmp
1.3G    /space/podman/volumes
35G     total

$ sudo podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         112         23          49.05GB     41.46GB (0%)
Containers     24          24          1.31MB      0B (0%)
Local Volumes  46          48          701MB       0B (0%)

… so podman's reported 37GB saving is actually more like 11/12GB? The "Images" 'reclaimable' figures from before and after don't seem to add up either. Having said this, even though I'm suspicious of 24 "Containers" having a size of 1.31MB, that figure is indeed 4.783MB - 3.472MB from the first system df.

Luap99 added a commit to Luap99/libpod that referenced this issue Mar 21, 2022
The calculate the percentage we need floating point numbers. The current
code however casted the result of reclaimable/size to an int first.
Casting to an int in go will just discard the decimal points, thus the
result was either 0 or 1 so if multiplied by 100 it would show up as 0%
or 100%.

To fix this we have to multiply by 100 first before casting the result
to an int. Also add a check for div by zero which results in NaN and use
math.Round() to correctly round a number.

Ref containers#13516

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
@Luap99
Copy link
Member

Luap99 commented Mar 21, 2022

#13575 fixes the percent calculation but it does not fix the incorrect values for the volume

mheon pushed a commit to mheon/libpod that referenced this issue Mar 30, 2022
The calculate the percentage we need floating point numbers. The current
code however casted the result of reclaimable/size to an int first.
Casting to an int in go will just discard the decimal points, thus the
result was either 0 or 1 so if multiplied by 100 it would show up as 0%
or 100%.

To fix this we have to multiply by 100 first before casting the result
to an int. Also add a check for div by zero which results in NaN and use
math.Round() to correctly round a number.

Ref containers#13516

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@cdoern
Copy link
Contributor

cdoern commented Jun 28, 2022

@rhatdan I think this is the issue that sally was describing to us a few weeks back. Seems eerily similar and concerning that the same thing is happening to two people. I will look at this.

@cdoern cdoern self-assigned this Jun 28, 2022
cdoern added a commit to cdoern/podman that referenced this issue Jun 28, 2022
currently, podman system df incorrectly calculates the reclaimable storage for
volumes, using a cumulative reclaimable variable that is incremented and placed into each
report entry causing values to rise above 100%.

Switch this variables to be in the context of the loop, so it resets per volume just like the size variable does.

resolves containers#13516

Signed-off-by: Charlie Doern <cdoern@redhat.com>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants