podman rmi gives "layer not known" when executed in parallel #6510
Comments
openshift-ci-robot
added
the
kind/bug
|
@nalind @vrothberg Does this look like an error where another process already removed a layer that we wanted removed? IE, something we can just ignore and continue running? |
|
A friendly reminder that this issue had no activity for 30 days. |
|
Any news on that? In the meanwhile we've updated to: The problem still exists. |
|
Apologies for not getting back to this issues earlier. I'll take a look right now. |
|
Hi, Same situation after last upgrade: |
|
@nalind was looking into the c/storage parts but we don't have news yet. We dedicate the entire next week to bug scrubbing. Maybe, we manage to get to the core of it. |
|
I'm going to close this ticket. |
|
I'm going to reopen - this is definitely still a problem, and we are still interested in solving it. |
|
@vrothberg Mind taking a look at this one? |
|
Pulling in @nalind as well who wanted to look into the c/storage parts. |
|
@nalind @rhatdan This script reproduces issue https://github.com/jwhonce/podman/blob/issues/6510/hack/build-abuse.sh |
|
Guys having alike issue, [root@overcloud-controller-0 ~]# podman ps -a | grep redis-bundle-podman-0 |
|
Here's another reproducer. I don't know if it's the same root cause (this one has to do with vfs and storage opts) but at least it's a 100%-reliable reproducer: #!/bin/bash
set -ex
PROOT=/dev/shm/mypodman
rm -rf $PROOT
mkdir -p $PROOT/root $PROOT/cache
IMAGE=quay.io/libpod/testimage:20200929
PODMAN="./bin/podman --root $PROOT/cache --storage-driver vfs"
$PODMAN pull $IMAGE
PODMAN="./bin/podman --storage-opt vfs.imagestore=$PROOT/cache --root $PROOT/root --storage-driver vfs"
$PODMAN pull $IMAGE
$PODMAN rmi -afThe last line will always fail with: |
|
A friendly reminder that this issue had no activity for 30 days. |
|
A friendly reminder that this issue had no activity for 30 days. |
vrothberg
added
the
In Progress
The c/storage library is subject to TOCTOUs as the central container and image storage may be shared by many instances of many tools. As shown in containers#6510, it's fairly easy to have multiple instances of Podman running in parallel and yield image-lookup errors when removing them. The underlying issue is the TOCTOU of removal being split into multiple stages of first reading the local images and then removing them. Some images may already have been removed in between the two stages. To make image removal more robust, handle errors at stage two when a given image is not present (anymore) in the storage. Fixes: containers#6510 Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The c/storage library is subject to TOCTOUs as the central container and image storage may be shared by many instances of many tools. As shown in containers#6510, it's fairly easy to have multiple instances of Podman running in parallel and yield image-lookup errors when removing them. The underlying issue is the TOCTOU of removal being split into multiple stages of first reading the local images and then removing them. Some images may already have been removed in between the two stages. To make image removal more robust, handle errors at stage two when a given image is not present (anymore) in the storage. Fixes: containers#6510 Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The following bugs seem to be fixed: * containers#6510 (e2e tests) - podman rmi gives "layer not known" * containers#9915 (buildah-bud tests) - podman build --arch * containers#10248 - spurious warning from first-ever invocation of rootless podman Signed-off-by: Ed Santiago <santiago@redhat.com>
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
We use a Jenkins pipeline with parallel stages. Each stage builds, tags, pushs and removes a container image. In about half of the pipeline runs the
rmicommand fails withlayer not known.We suspect a race condition during the parallel runs.
Running the same pipeline on "older" centos7 machines with docker-ce installed works without any problem.
In the pipeline script the command
dockeris used to be compatible with the "old" workers. On the actual workersdockeris a symlink to thepodmanexecutable.Many thanks for your help!
Steps to reproduce the issue:
The original pipeline is more complex with 7 images build in parallel. It was stripped down for debugging and the problem was reproduced with this version:
Describe the results you received:
Describe the results you expected:
Images should be untagged / removed.
Additional information you deem important (e.g. issue happens only occasionally):
The same problem occurs on CentOS 8 workers with
container-tools:rhel8installed.Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Additional environment details (AWS, VirtualBox, physical, etc.):
Physical machine:
The text was updated successfully, but these errors were encountered: