podman rm sometimes fails with "no such file or directory"

[hardys]

Sometimes 04_setup_ironic.sh fails like this:

+ sudo podman pod exists ironic-pod
+ sudo podman pod rm ironic-pod -f
Error: no such file or directory
make: *** [ironic] Error 125

The problem is that podman pod exists returns zero, but then the rm fails, this seems to be some podman DB corruption, exact cause unclear but in some cases it seems related to unplanned reboots.

We need to find/file an upstream bug for this, but until we find the root cause and fix, there are two known workarounds:

The simplest but most invasive fix is to remove the containers dir and reinstall podman, but note this will require re-downloading all images:

make clean && rm -fr /var/lib/containers && yum reinstall podman

The second option is to do go get github.com/br0xen/boltbrowser
then sudo ${HOME}/go/bin/boltbrowser /var/lib/containers/storage/libpod/bolt_state.db
and remove all the references to the pod and to the container interactively in that db (thanks to @celebdor for the tip on this one)

[hardys]

containers/podman#2900 may be related

[hogepodge]

As far as I can tell, this error happens because the script uses a force shutdown. I ran into this issue three times over the last few week, and removing the force flags has apparently resolved this for me.

[dantrainor]

I'm running in to this again today. The workaround seemed to have worked, but probably isn't a long-term solution.

[hardys]

As far as I can tell, this error happens because the script uses a force shutdown. I ran into this issue three times over the last few week, and removing the force flags has apparently resolved this for me.

Any more information on why you think this could corrupt the state of podman managing the containers, vs only the workload itself?

[russellb]

As far as I can tell, this error happens because the script uses a force shutdown. I ran into this issue three times over the last few week, and removing the force flags has apparently resolved this for me.

Any more information on why you think this could corrupt the state of podman managing the containers, vs only the workload itself?

I don't have an explanation, but this same problem happened several times in the open infra summit demo environment, and was resolved the same way, by removing the force option.

[dantrainor]

I ran in to this a few times this morning, as well. Interesting to note that a manual 'sudo podman rm ironic -f' works without error, post-failure. Think it's a timing issue? Not being able to remove a pod that's in a not-fully-started state or some such? Sorry if my verbiage is a little off, showing my podman inexperience here.

[hardys]

Ok so it seems this will be solved by two bugfixes to podman, one has landed, one is currently under review ref containers/podman#2900 (comment)

We can leave this issue open until we can rebase on a newer version that fixes the issue

[honza]

This started happening to me yesterday. It happens during every run.

[dhellmann]

It looks like the libpod ticket has been closed.

Is this still an issue?

[hogepodge]

Not for me. I would still be wary of using force, but that's more because it seems to be less safe by its very nature than any outstanding issue.

[dhellmann]

OK, I'll close this out so we can clean up our backlog. Thanks, @hogepodge !