cannot remove container [...]: given PIDs did not die within timeout

[edsantiago]

New flake, failed twice in one PR:

not ok 137 podman container rm --force doesn't leave running processes
...
# podman-remote --url unix:/tmp/podman_tmp_i9cG rm -f cTG9q6WAKk9c0Bvs343yuzK8rbPiBTs
Error: cannot remove container <sha> as it could not be stopped: given PIDs did not die within timeou

• f37 root, remote
• f37 root aarch64

[vrothberg]

Currently, the timeout is 5 seconds. That may not be enough on very busy nodes. In CI, 20 seconds seem to be the current magic limit. Shall we bump it to that?

@giuseppe @mheon WDYT?

[mheon]

I'm a little reluctant here, because it will basically result in a frozen Podman for most of a minute if the process is legitimately not going to die (zombie process or similar) - and I am somehow doubting that the CI VMs are slow enough that it takes more than 5 seconds for a SIGKILL to be processed (but then again, maybe I'm wrong).

[vrothberg]

There may very well be (another) bug? Let's keep an eye open and wait for more reports from @edsantiago. CI is sick at the moment, so there may be other things interfering (although unlikely).

[edsantiago]

Uh-oh

not ok 521 podman kube play - multi-pod YAML
...
        # $ podman pod stop pod1 pod2
         # 3878f14ab92abfba94526a0df961abce9183781525442b4906c4f2103009af14
         # 7568d3ac6f56544faabafea7c984e568f401122e9031c43c0c75b6bf82de09a3
         # open pidfd: No such process
         # time="2023-01-18T08:24:09-06:00" level=error msg="Stopping service container 7da6d2bccb1d253b273ee65a083145215dabd290115edee7f8a0025dfa88465a: given PIDs did not die within timeout"
         # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
         # #|     FAIL: podman pod stop pod1 pod2
         # #| expected: !~ 'Stopping'
         # #|   actual:    '3878f14ab92abfba94526a0df961abce9183781525442b4906c4f2103009af14'
         # #|         >    '7568d3ac6f56544faabafea7c984e568f401122e9031c43c0c75b6bf82de09a3'
         # #|         >    'open pidfd: No such process'
         # #|         >    'time="2023-01-18T08:24:09-06:00" level=error msg="Stopping service container 7da6d2bccb1d253b273ee65a083145215dabd290115edee7f8a0025dfa88465a: given PIDs did not die within timeout"'
         # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[vrothberg]

I'll tackle this one tomorrow. I had a look today already but then moved on to the bud flake

[vrothberg]

it smells so badly like a race

[vrothberg]

it smells so badly like a race

#17165 ... same race as before but an improvement of the previous PR (details in the PR description)