podman stop: rootless netns ref counter out of sync, counter is at -1, resetting it back to 0 #21569
Assignees
Labels
flakes
Flakes from Continuous Integration
locked - please file new issue/PR
Assist humans wanting to comment on an old issue or PR with locked comments.
rootless
Comments
|
This error is new with my c/common network rework. Although I am very sure the bug is not there, I suspect for a very long time (also hinting from other flakes) that it is possible that we cleanup twice. https://api.cirrus-ci.com/v1/artifact/task/6507554541404160/html/int-podman-fedora-39-rootless-host-sqlite.log.html#t--Podman-kube-play-test-with-reserved-init-annotation-in-yaml--1 |
|
All quiet since initial filing, but just saw this one, f38. The "No such device" error is new: |
|
yeah once I have a bit more time I give you some diff for your flake retry PR to capture the full stack traces from within the rootless netns code to see how we end up there in cleanup twice by adding a ton of debugging code that should not be merged into main. |
|
Checking in. Still rootless-only. Happening even with last week's VMs (pasta 2024-03-20).
|
|
Ping, hi. Here's one with two failure messages: The real failure is probably "unable to clean up network", which could be #19721 or could be yet another something new. Current flake list (may be incomplete, because some of these may be filed under other issues)
|
|
The real error is something calls network cleanup twice AFAICT, which also applies to ##19721 Any I really need full stack traces of both process calling cleanup but there is no good way of getting this as likely only one of them errors out like that so I cannot include in the error message. |
|
This one is really blowing up lately. Maybe the new CI VMs, or maybe the tmpfs PR, or maybe something else entirely. Last 30 days:
|
|
This one always seems to happen along with "Unable to clean up network" (#19721). For example: Are they the same bug? Should I close one in favor of the other, and assign them all to the same issue?
|
|
My guess is that they have the same cause, that said I also don't understand what is going on other than we are trying to cleanup the networks twice so I cannot be sure about that. Anyhow looks like this has gotten much worse so I put this on my priority list for next week. |
|
Still happening frequently, even in non-Ed PRs. Here's one new error message I hadn't seen before (the middle one, aardvark netns): This was in my parallel-bats PR |
|
I have been looking at the errors for hours today without having any solid idea what is causing this stuff. I found one issue around But this is not directly related to the symptoms here. It is very clear from the error sequence that something is calling cleanupNetwork() twice but this should practically be impossible unless the cleanup process was killed after it called netavark but before it wrote the netns to the db (I don't see that happening though). Your new error looks even weirder. Because netavark tries to delete the veth interface first (and there is no error logged about it so it worked) and only then the bridge (which failed). |
|
It would be great if we manage to get a local reproducer, I tried running the commands from the failed tests without luck. |
|
It only fails in kube-related tests. Eliminating int tests and my bats-parallel PR, I see failures only in |
|
That sounds like a good idea |
|
Well, that was useless: Looking at the journal (search in-page for ...that is, shut down veth0 before the pod-rm. And some systemd stuff nearby. Is it possible that systemd is engaging in some unwanted cleanup/skullduggery? |
|
well cleanup happens when the ctr stops not when it is removed so it is expected that interfaces are removed before the actual container/pod is removed. What is interesting here is to look for The important bit here is that there is no cleanup event after it got restarted and killed. Generally we expect But in this case there is no cleanup event at all suggesting that something went wrong during cleanup (possibly the process was killed or we hit some error but neither is logged anywhere in the journal so it is impossible to know) |
Luap99
added a commit
to Luap99/libpod
that referenced
this issue
Jul 30, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
Reproducer! Not a great one, complicated to set up yadda yadda, but here goes:
$ while :;do hack/bats --rootless 700:service-conta || break;doneReproduces the Then instrument podman to your heart's content. HTH |
Luap99
added a commit
to Luap99/libpod
that referenced
this issue
Jul 31, 2024
When using service containers and play kube we create a complicated set of dependencies. First in a pod all conmon/container cgroups are part of one slice, that slice will be removed when the entire pod is stopped resulting in systemd killing all processes that were part in it. Now the issue here is around the working of stopPodIfNeeded() and stopIfOnlyInfraRemains(), once a container is cleaned up it will check if the pod should be stopped depending on the pod ExitPolicy. If this is the case it wil stop all containers in that pod. However in our flaky test we calle podman pod kill which logically killed all containers already. Thus the logic now thinks on cleanup it must stop the pod and calls into pod.stopWithTimeout(). Then there we try to stop but because all containers are already stopped it just throws errors and never gets to the point were it would call Cleanup(). So the code does not do cleanup and eventually calls removePodCgroup() which will cause all conmon and other podman cleanup processes of this pod to be killed. Thus the podman container cleanup process was likely killed while actually trying to the the proper cleanup which leaves us in a bad state. Following commands such as podman pod rm will try to the cleanup again as they see it was not completed but then fail as they are unable to recover from the partial cleanup state. Long term network cleanup needs to be more robust and ideally should be idempotent to handle cases were cleanup was killed in the middle. Fixes containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
mheon
pushed a commit
to mheon/libpod
that referenced
this issue
Jul 31, 2024
When using service containers and play kube we create a complicated set of dependencies. First in a pod all conmon/container cgroups are part of one slice, that slice will be removed when the entire pod is stopped resulting in systemd killing all processes that were part in it. Now the issue here is around the working of stopPodIfNeeded() and stopIfOnlyInfraRemains(), once a container is cleaned up it will check if the pod should be stopped depending on the pod ExitPolicy. If this is the case it wil stop all containers in that pod. However in our flaky test we calle podman pod kill which logically killed all containers already. Thus the logic now thinks on cleanup it must stop the pod and calls into pod.stopWithTimeout(). Then there we try to stop but because all containers are already stopped it just throws errors and never gets to the point were it would call Cleanup(). So the code does not do cleanup and eventually calls removePodCgroup() which will cause all conmon and other podman cleanup processes of this pod to be killed. Thus the podman container cleanup process was likely killed while actually trying to the the proper cleanup which leaves us in a bad state. Following commands such as podman pod rm will try to the cleanup again as they see it was not completed but then fail as they are unable to recover from the partial cleanup state. Long term network cleanup needs to be more robust and ideally should be idempotent to handle cases were cleanup was killed in the middle. Fixes containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Oct 22, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Oct 23, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Oct 25, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Oct 28, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Oct 31, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Nov 4, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Nov 4, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Nov 6, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Nov 6, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Nov 7, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Nov 7, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this issue
Nov 7, 2024
In theory when syslog is set the cleanup process should log its errors to syslog (journald) so we can have a look at the errors in CI. Without it podman container cleanup errors will never be logged anywhere. In order to rey to debug containers#21569 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
stale-locking-app
bot
added
the
locked - please file new issue/PR
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Seen mostly in system tests, BUT ALSO IN E2E TESTS WHERE IT IS IMPOSSIBLE TO DETECT. Because it happens in test cleanup, where
rmdoes not trigger a test failure. So I caught all these by accident. THIS IS HAPPENING A LOT MORE THAN WE ARE AWARE OF.The text was updated successfully, but these errors were encountered: