Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

couldn't create rootless container: "mounting storage for container ... input/output error" #21274

Closed
TomasTomecek opened this issue Jan 17, 2024 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@TomasTomecek
Copy link
Contributor

TomasTomecek commented Jan 17, 2024

Issue Description

What a heisenbug!

I created this bugreport with all the info and tried to reproduce it one last time and study logs in journald... But suddenly the error disappeared, my container was created and everything worked normally.

What can I do in the future if I stumble into this 'input/output error'?

Steps to reproduce the issue

Steps to reproduce the issue

  1. pull the image (it's pretty big)
$ podman pull docker.io/atinoda/text-generation-webui:llama-cpu-nightly
Trying to pull docker.io/atinoda/text-generation-webui:llama-cpu-nightly...
Getting image source signatures
Copying blob 5e5846364eee skipped: already exists
Copying blob 45f7ea5367fe skipped: already exists
Copying blob 3d97a47c3c73 skipped: already exists
Copying blob 12cd4d19752f skipped: already exists
Copying blob da5a484f9d74 skipped: already exists
Copying blob fd355de1d1f2 skipped: already exists
Copying blob 3480bb79c638 skipped: already exists
Copying blob e7016935dd60 skipped: already exists
Copying blob 99541166a133 skipped: already exists
Copying blob 515cf72a037d skipped: already exists
Copying blob 8999112df5b0 skipped: already exists
Copying blob 9d852da795c6 skipped: already exists
Copying blob e7509cb2ff9d skipped: already exists
Copying blob 814fabdda473 skipped: already exists
Copying blob 1270508ad9c5 skipped: already exists
Copying blob a2f44eb18719 skipped: already exists
Copying blob 4f4fb700ef54 skipped: already exists
Copying blob aece8493d397 skipped: already exists
Copying blob 5923630bdcf7 skipped: already exists
Copying blob 53fb119af535 skipped: already exists
Copying blob 3f27dc1984d3 skipped: already exists
Copying blob 765c2b3dd8a2 skipped: already exists
Copying blob 4f4fb700ef54 skipped: already exists
Copying blob 4860f838e07b skipped: already exists
Copying blob 35889fc87a4d skipped: already exists
Copying config 377e35f1cd done   |
Writing manifest to image destination
377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5
  1. run it:
$ podman run --rm -ti docker.io/atinoda/text-generation-webui:llama-cpu-nightly
ERRO[0000] Unmounting /home/tt/.local/share/containers/storage/overlay/a65985976a325b1406893e4090a805a4d4a140cc6c3d6b864eb5d5e39894721f/merged: invalid argument
Error: mounting storage for container 2439c64aa6b1144934bab15cd2e5f9e1896575f446977a46868eda8c7004b25f: creating overlay mount to /home/tt/.local/share/containers/storage/overlay/a65985976a325b1406893e4090a805a4d4a140cc6c3d6b864eb5d5e39894721f/merged, mount_data="lowerdir=/home/tt/.local/share/
containers/storage/overlay/l/YKPB3N76EBZZX3S2Q77U66STAV:/home/tt/.local/share/containers/storage/overlay/l/KLVOPTRRTJ4YIKN2K52LGTHABK:/home/tt/.local/share/containers/storage/overlay/l/ZEPR53A7FSWFTDH3KRT3YZUYBA:/home/tt/.local/share/containers/storage/overlay/l/CDKKVI3IAWLX5TR2GR24Z2N34N:/home
/tt/.local/share/containers/storage/overlay/l/NKUXGMEVK34YEKU7NVVZBZVN45:/home/tt/.local/share/containers/storage/overlay/l/7BZZIX4JS4GJPUXVTRLHRH2VXL:/home/tt/.local/share/containers/storage/overlay/l/LBAVIOJPGUS5ZCT3ZJ7ONJB7ZW:/home/tt/.local/share/containers/storage/overlay/l/IWCPHJXASWQ2NSH
BO6FSHESY2C:/home/tt/.local/share/containers/storage/overlay/l/2K4J54HLEUHFQOKVYLE5QKU2OR:/home/tt/.local/share/containers/storage/overlay/l/S4WE2CR4U6Q7I3TDYVZEUSM6XZ:/home/tt/.local/share/containers/storage/overlay/l/CENEVQNAHNJR7MGAARV2UZENKH:/home/tt/.local/share/containers/storage/overlay/
l/FULIC5K7URPATHRMJU72WCY4J3:/home/tt/.local/share/containers/storage/overlay/l/SMW6UCAXUA5X4ERVY7CRGKXM55:/home/tt/.local/share/containers/storage/overlay/l/EQVPPTDH5DMH6ZEOWJDMQSNGYV:/home/tt/.local/share/containers/storage/overlay/l/PXMO7ZKSHFOR5LIQXHMVBZZUVT:/home/tt/.local/share/containers
/storage/overlay/l/ISFE7DXZW5UAVBFNRS5IRUFG36:/home/tt/.local/share/containers/storage/overlay/l/36JY74YEQT57XSCXQLEZ4SXYR4:/home/tt/.local/share/containers/storage/overlay/l/GXHVJMCLBU7YZTZORG3TKBKEME:/home/tt/.local/share/containers/storage/overlay/l/BTPGKSPW32LBF7NP66P5G4G4GD:/home/tt/.local
/share/containers/storage/overlay/l/U732HZXDQTM63CT7BC22VME5FN:/home/tt/.local/share/containers/storage/overlay/l/5UPWF7RFLF35NLPZEE2SHAOJZC:/home/tt/.local/share/containers/storage/overlay/l/GOZRSDOKHBQEJMI7AU26MNPRKO:/home/tt/.local/share/containers/storage/overlay/l/JC3ITBCTO3BSDYIP2SEWYOTMN
B:/home/tt/.local/share/containers/storage/overlay/l/HDX3WSF6QU7JOBAFIZKCRYLAX6:/home/tt/.local/share/containers/storage/overlay/l/E4LSVS6SM4HXUHX774VG3ZYDYU,upperdir=/home/tt/.local/share/containers/storage/overlay/a65985976a325b1406893e4090a805a4d4a140cc6c3d6b864eb5d5e39894721f/diff,workdir=/
home/tt/.local/share/containers/storage/overlay/a65985976a325b1406893e4090a805a4d4a140cc6c3d6b864eb5d5e39894721f/work,userxattr,volatile,context=\"system_u:object_r:container_file_t:s0:c380,c424\"": input/output error

Describe the results you received

Here's the output with --log-level=debug

DEBU[0000] Looking up image "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" ...
DEBU[0000] parsed reference into "[overlay@/home/tt/.local/share/containers/storage+/run/user/1000/containers]@377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5"
DEBU[0000] Found image "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" as "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" in local containers storage
DEBU[0000] Found image "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" as "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" in local containers storage ([overlay@/home/tt/.local/share/containers/storage+/run/user/1000/containers]@377e35f1cd04b77f8ab98e42d52d81b64ded8172c53
f1dd3126a3b1c59fbb5b5)
DEBU[0000] exporting opaque data as blob "sha256:377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5"
DEBU[0000] Looking up image "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" ...
DEBU[0000] parsed reference into "[overlay@/home/tt/.local/share/containers/storage+/run/user/1000/containers]@377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5"
DEBU[0000] Found image "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" as "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" in local containers storage
DEBU[0000] Found image "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" as "docker.io/atinoda/text-generation-webui:llama-cpu-nightly" in local containers storage ([overlay@/home/tt/.local/share/containers/storage+/run/user/1000/containers]@377e35f1cd04b77f8ab98e42d52d81b64ded8172c53
f1dd3126a3b1c59fbb5b5)
DEBU[0000] exporting opaque data as blob "sha256:377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5"
DEBU[0000] Inspecting image 377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5
DEBU[0000] exporting opaque data as blob "sha256:377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5"
DEBU[0000] Inspecting image 377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5
DEBU[0000] Inspecting image 377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5
DEBU[0000] Inspecting image 377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Allocated lock 71 for container 2440c80ad59766da75059ee9edf2488fd56581927e06382e822b956c82d5d989
DEBU[0000] parsed reference into "[overlay@/home/tt/.local/share/containers/storage+/run/user/1000/containers]@377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5"
DEBU[0000] exporting opaque data as blob "sha256:377e35f1cd04b77f8ab98e42d52d81b64ded8172c53f1dd3126a3b1c59fbb5b5"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "2440c80ad59766da75059ee9edf2488fd56581927e06382e822b956c82d5d989"
DEBU[0000] Container "2440c80ad59766da75059ee9edf2488fd56581927e06382e822b956c82d5d989" has work directory "/home/tt/.local/share/containers/storage/overlay-containers/2440c80ad59766da75059ee9edf2488fd56581927e06382e822b956c82d5d989/userdata"
DEBU[0000] Container "2440c80ad59766da75059ee9edf2488fd56581927e06382e822b956c82d5d989" has run directory "/run/user/1000/containers/overlay-containers/2440c80ad59766da75059ee9edf2488fd56581927e06382e822b956c82d5d989/userdata"
DEBU[0000] Handling terminal attach
INFO[0000] Received shutdown.Stop(), terminating!        PID=933148
DEBU[0000] Enabling signal proxying
DEBU[0000] Made network namespace at /run/user/1000/netns/netns-36ca2e01-72e6-4c72-1b95-d419a0837942 for container 2440c80ad59766da75059ee9edf2488fd56581927e06382e822b956c82d5d989
DEBU[0000] Cached value indicated that volatile is being used
DEBU[0000] overlay: mount_data=lowerdir=/home/tt/.local/share/containers/storage/overlay/l/YKPB3N76EBZZX3S2Q77U66STAV:/home/tt/.local/share/containers/storage/overlay/l/KLVOPTRRTJ4YIKN2K52LGTHABK:/home/tt/.local/share/containers/storage/overlay/l/ZEPR53A7FSWFTDH3KRT3YZUYBA:/home/tt/.local/share
/containers/storage/overlay/l/CDKKVI3IAWLX5TR2GR24Z2N34N:/home/tt/.local/share/containers/storage/overlay/l/NKUXGMEVK34YEKU7NVVZBZVN45:/home/tt/.local/share/containers/storage/overlay/l/7BZZIX4JS4GJPUXVTRLHRH2VXL:/home/tt/.local/share/containers/storage/overlay/l/LBAVIOJPGUS5ZCT3ZJ7ONJB7ZW:/hom
e/tt/.local/share/containers/storage/overlay/l/IWCPHJXASWQ2NSHBO6FSHESY2C:/home/tt/.local/share/containers/storage/overlay/l/2K4J54HLEUHFQOKVYLE5QKU2OR:/home/tt/.local/share/containers/storage/overlay/l/S4WE2CR4U6Q7I3TDYVZEUSM6XZ:/home/tt/.local/share/containers/storage/overlay/l/CENEVQNAHNJR7M
GAARV2UZENKH:/home/tt/.local/share/containers/storage/overlay/l/FULIC5K7URPATHRMJU72WCY4J3:/home/tt/.local/share/containers/storage/overlay/l/SMW6UCAXUA5X4ERVY7CRGKXM55:/home/tt/.local/share/containers/storage/overlay/l/EQVPPTDH5DMH6ZEOWJDMQSNGYV:/home/tt/.local/share/containers/storage/overlay
/l/PXMO7ZKSHFOR5LIQXHMVBZZUVT:/home/tt/.local/share/containers/storage/overlay/l/ISFE7DXZW5UAVBFNRS5IRUFG36:/home/tt/.local/share/containers/storage/overlay/l/36JY74YEQT57XSCXQLEZ4SXYR4:/home/tt/.local/share/containers/storage/overlay/l/GXHVJMCLBU7YZTZORG3TKBKEME:/home/tt/.local/share/container
s/storage/overlay/l/BTPGKSPW32LBF7NP66P5G4G4GD:/home/tt/.local/share/containers/storage/overlay/l/U732HZXDQTM63CT7BC22VME5FN:/home/tt/.local/share/containers/storage/overlay/l/5UPWF7RFLF35NLPZEE2SHAOJZC:/home/tt/.local/share/containers/storage/overlay/l/GOZRSDOKHBQEJMI7AU26MNPRKO:/home/tt/.loca
l/share/containers/storage/overlay/l/JC3ITBCTO3BSDYIP2SEWYOTMNB:/home/tt/.local/share/containers/storage/overlay/l/HDX3WSF6QU7JOBAFIZKCRYLAX6:/home/tt/.local/share/containers/storage/overlay/l/E4LSVS6SM4HXUHX774VG3ZYDYU,upperdir=/home/tt/.local/share/containers/storage/overlay/6d6faee72321d9896
98b7b9c24e5baa4cce03d2586f54173b80573d10676c2a6/diff,workdir=/home/tt/.local/share/containers/storage/overlay/6d6faee72321d989698b7b9c24e5baa4cce03d2586f54173b80573d10676c2a6/work,userxattr,volatile,context="system_u:object_r:container_file_t:s0:c389,c604"
ERRO[0000] Unmounting /home/tt/.local/share/containers/storage/overlay/6d6faee72321d989698b7b9c24e5baa4cce03d2586f54173b80573d10676c2a6/merged: invalid argument
DEBU[0000] Failed to mount container "2440c80ad59766da75059ee9edf2488fd56581927e06382e822b956c82d5d989": creating overlay mount to /home/tt/.local/share/containers/storage/overlay/6d6faee72321d989698b7b9c24e5baa4cce03d2586f54173b80573d10676c2a6/merged, mount_data="lowerdir=/home/tt/.local/share
/containers/storage/overlay/l/YKPB3N76EBZZX3S2Q77U66STAV:/home/tt/.local/share/containers/storage/overlay/l/KLVOPTRRTJ4YIKN2K52LGTHABK:/home/tt/.local/share/containers/storage/overlay/l/ZEPR53A7FSWFTDH3KRT3YZUYBA:/home/tt/.local/share/containers/storage/overlay/l/CDKKVI3IAWLX5TR2GR24Z2N34N:/hom
e/tt/.local/share/containers/storage/overlay/l/NKUXGMEVK34YEKU7NVVZBZVN45:/home/tt/.local/share/containers/storage/overlay/l/7BZZIX4JS4GJPUXVTRLHRH2VXL:/home/tt/.local/share/containers/storage/overlay/l/LBAVIOJPGUS5ZCT3ZJ7ONJB7ZW:/home/tt/.local/share/containers/storage/overlay/l/IWCPHJXASWQ2NS
HBO6FSHESY2C:/home/tt/.local/share/containers/storage/overlay/l/2K4J54HLEUHFQOKVYLE5QKU2OR:/home/tt/.local/share/containers/storage/overlay/l/S4WE2CR4U6Q7I3TDYVZEUSM6XZ:/home/tt/.local/share/containers/storage/overlay/l/CENEVQNAHNJR7MGAARV2UZENKH:/home/tt/.local/share/containers/storage/overlay
/l/FULIC5K7URPATHRMJU72WCY4J3:/home/tt/.local/share/containers/storage/overlay/l/SMW6UCAXUA5X4ERVY7CRGKXM55:/home/tt/.local/share/containers/storage/overlay/l/EQVPPTDH5DMH6ZEOWJDMQSNGYV:/home/tt/.local/share/containers/storage/overlay/l/PXMO7ZKSHFOR5LIQXHMVBZZUVT:/home/tt/.local/share/container
s/storage/overlay/l/ISFE7DXZW5UAVBFNRS5IRUFG36:/home/tt/.local/share/containers/storage/overlay/l/36JY74YEQT57XSCXQLEZ4SXYR4:/home/tt/.local/share/containers/storage/overlay/l/GXHVJMCLBU7YZTZORG3TKBKEME:/home/tt/.local/share/containers/storage/overlay/l/BTPGKSPW32LBF7NP66P5G4G4GD:/home/tt/.loca
l/share/containers/storage/overlay/l/U732HZXDQTM63CT7BC22VME5FN:/home/tt/.local/share/containers/storage/overlay/l/5UPWF7RFLF35NLPZEE2SHAOJZC:/home/tt/.local/share/containers/storage/overlay/l/GOZRSDOKHBQEJMI7AU26MNPRKO:/home/tt/.local/share/containers/storage/overlay/l/JC3ITBCTO3BSDYIP2SEWYOTM
NB:/home/tt/.local/share/containers/storage/overlay/l/HDX3WSF6QU7JOBAFIZKCRYLAX6:/home/tt/.local/share/containers/storage/overlay/l/E4LSVS6SM4HXUHX774VG3ZYDYU,upperdir=/home/tt/.local/share/containers/storage/overlay/6d6faee72321d989698b7b9c24e5baa4cce03d2586f54173b80573d10676c2a6/diff,workdir=
/home/tt/.local/share/containers/storage/overlay/6d6faee72321d989698b7b9c24e5baa4cce03d2586f54173b80573d10676c2a6/work,userxattr,volatile,context=\"system_u:object_r:container_file_t:s0:c389,c604\"": input/output error

The input/output error and /home/tt/.local/share/containers/storage/overlay/6d6faee72321d989698b7b9c24e5baa4cce03d2586f54173b80573d10676c2a6/merged: invalid argument are highly suspicious though they don't exactly say what's wrong.

I studied the rootless troubleshooting guide and none of the advises for 'invalid argument' apply to me. I have been using this rootless setup for years without a problem.

Describe the results you expected

Running container

podman info output

host: 
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 47.87
    systemPercent: 11.08
    userPercent: 41.05
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "39"
  eventLogger: journald
  freeLocks: 1977
  hostname: cashew
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.4-200.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2144911360
  memTotal: 33130041344
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
    package: containernetworking-plugins-1.3.0-3.fc39.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: crun
    package: crun-1.12-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231204.gb86afe3-1.fc39.x86_64
    version: |
      pasta 0^20231204.gb86afe3-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 6576758784
  swapTotal: 8589930496
  uptime: 364h 16m 51.00s (Approximately 15.17 days)
  variant: ""
store:
  configFile: /home/tt/.config/containers/storage.conf
  containerStore:
    number: 67
    paused: 0
    running: 1
    stopped: 66
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/tt/.local/share/containers/storage
  graphRootAllocated: 254339448832
  graphRootUsed: 236537688064
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 130
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/tt/.local/share/containers/storage/volumes
version:
  APIVersion: 4.8.3
  Built: 1704291100
  BuiltTime: Wed Jan  3 15:11:40 2024
  GitCommit: ""
  GoVersion: go1.21.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

@TomasTomecek TomasTomecek added the kind/bug Categorizes issue or PR as related to a bug. label Jan 17, 2024
@TomasTomecek TomasTomecek changed the title couldn't create user container: "mounting storage for container ... input/output error" couldn't create rootless container: "mounting storage for container ... input/output error" Jan 17, 2024
@rhatdan
Copy link
Member

rhatdan commented Jan 17, 2024

The input/output error and /home/tt/.local/share/containers/storage/overlay/6d6faee72321d989698b7b9c24e5baa4cce03d2586f54173b80573d10676c2a6/merged: invalid argument are highly suspicious though they don't exactly say what's wrong.

We really should display the mount options when we get a failure.

@giuseppe
Copy link
Member

I've tried locally and I am not able to reproduce, could you run the following command and share the output.log file?

podman unshare strace -f -v -s 1000 -o output.log podman run --rm -ti docker.io/atinoda/text-generation-webui:llama-cpu-nightly

is there any useful information in the dmesg output?

@TomasTomecek
Copy link
Contributor Author

Thanks for the guidance, @giuseppe and @rhatdan

As I said in the original post, once I started debugging the issue, it went away and the container was created and started correctly.

If it ever happens again, I'll provide strace and dmesg logs. Thanks!

@TomasTomecek TomasTomecek closed this as not planned Won't fix, can't repro, duplicate, stale Jan 18, 2024
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Apr 18, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 18, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

3 participants