Podman: "State 'stop-sigterm' timed out. Killing."

[sokai]

Hey community!

Is anybody else experiencing problems during stoping/restarting the Homarr container with Podman?
I use Nixos and have set-up a very simple OCI container (virtualisation.oci-containers.containers) that runs with Podman with systemd.

(I ask here because I couldn't find any hints in the issues.)
#ty + KR

Here you can see the (redacted) log for a restart by systemd (sudo systemctl restart podman-homarr.service). After Podman killed the service it comes up again and runs as normal …

Mar 28 21:18:52 host sudo[1392616]:    user : TTY=pts/2 ; PWD=/home/user/Code/Git/user/nixos/nixos-host ; USER=root ; COMMAND=/run/current-system/sw/bin/systemctl restart podman-homarr.service
Mar 28 21:18:52 host systemd[1]: Stopping podman-homarr.service...
Mar 28 21:20:52 host systemd[1]: podman-homarr.service: State 'stop-sigterm' timed out. Killing.
Mar 28 21:20:52 host systemd[1]: podman-homarr.service: Killing process 1245947 (conmon) with signal SIGKILL.
Mar 28 21:20:52 host systemd[1]: podman-homarr.service: Main process exited, code=killed, status=9/KILL
Mar 28 21:21:02 host podman-homarr-post-stop[1392760]: time="2024-03-28T21:21:02+01:00" level=warning msg="StopSignal SIGTERM failed to stop container homarr in 10 seconds, resorting to SIGKILL"
Mar 28 21:21:02 host podman[1392760]: 2024-03-28 21:21:02.396226925 +0100 CET m=+10.066808311 container stop 69e088ff73b79d222acfc59cd1205b087ce36f3f591403116903c164a8ab6f12 (image=ghcr.io/ajnart/homarr:latest, name=homarr, org.opencontainers.image.url=https://github.com/ajnart/homarr, PODMAN_SYSTEMD_UNIT=podman-homarr.service, org.opencontainers.image.description=Customizable browser's home page to interact with your homeserver's Docker containers (e.g. Sonarr/Radarr), org.opencontainers.image.version=0.15.2, org.opencontainers.image.revision=5875b1b264c09b362705a1e14cebecf71b12e8ef, org.opencontainers.image.created=2024-03-15T18:49:20.269Z, org.opencontainers.image.licenses=MIT, org.opencontainers.image.title=homarr, org.opencontainers.image.source=https://github.com/ajnart/homarr)
Mar 28 21:21:02 host podman[1392760]: 2024-03-28 21:21:02.497997075 +0100 CET m=+10.168578460 container remove 69e088ff73b79d222acfc59cd1205b087ce36f3f591403116903c164a8ab6f12 (image=ghcr.io/ajnart/homarr:latest, name=homarr, org.opencontainers.image.description=Customizable browser's home page to interact with your homeserver's Docker containers (e.g. Sonarr/Radarr), org.opencontainers.image.version=0.15.2, org.opencontainers.image.revision=5875b1b264c09b362705a1e14cebecf71b12e8ef, org.opencontainers.image.source=https://github.com/ajnart/homarr, org.opencontainers.image.url=https://github.com/ajnart/homarr, PODMAN_SYSTEMD_UNIT=podman-homarr.service, org.opencontainers.image.created=2024-03-15T18:49:20.269Z, org.opencontainers.image.licenses=MIT, org.opencontainers.image.title=homarr)
Mar 28 21:21:02 host podman-homarr-post-stop[1392760]: 69e088ff73b79d222acfc59cd1205b087ce36f3f591403116903c164a8ab6f12
Mar 28 21:21:02 host systemd[1]: podman-homarr.service: Failed with result 'timeout'.
Mar 28 21:21:02 host systemd[1]: Stopped podman-homarr.service.
Mar 28 21:21:02 host systemd[1]: podman-homarr.service: Consumed 809ms CPU time, read 1.7M from disk, written 2.1M to disk, received 45.3K IP traffic, sent 7.2K IP traffic.
Mar 28 21:21:02 host systemd[1]: Starting podman-homarr.service...
Mar 28 21:21:02 host podman-homarr-start[1392826]: Trying to pull ghcr.io/ajnart/homarr:latest...

And this the restart with Podman (sudo podman restart 532ae295023a). It also seems to kill some processes, but the restart is much faster …

Mar 28 21:44:14 host podman[1400301]: 2024-03-28 21:44:14.51431891 +0100 CET m=+0.024350955 container restart 532ae295023a3425eb458d43bc24ab5f7d87529a60e3f7fc2872220b4dae0fd0 (image=ghcr.io/ajnart/homarr:latest, name=homarr, org.opencontainers.image.version=0.15.2, org.opencontainers.image.created=2024-03-15T18:49:20.269Z, org.opencontainers.image.licenses=MIT, PODMAN_SYSTEMD_UNIT=podman-homarr.service, org.opencontainers.image.url=https://github.com/ajnart/homarr, org.opencontainers.image.source=https://github.com/ajnart/homarr, org.opencontainers.image.description=Customizable browser's home page to interact with your homeserver's Docker containers (e.g. Sonarr/Radarr), org.opencontainers.image.title=homarr, org.opencontainers.image.revision=5875b1b264c09b362705a1e14cebecf71b12e8ef)
Mar 28 21:44:24 host podman[1400301]: 2024-03-28 21:44:24.551467298 +0100 CET m=+10.061499365 container died 532ae295023a3425eb458d43bc24ab5f7d87529a60e3f7fc2872220b4dae0fd0 (image=ghcr.io/ajnart/homarr:latest, name=homarr, org.opencontainers.image.created=2024-03-15T18:49:20.269Z, org.opencontainers.image.licenses=MIT, org.opencontainers.image.revision=5875b1b264c09b362705a1e14cebecf71b12e8ef, org.opencontainers.image.source=https://github.com/ajnart/homarr, PODMAN_SYSTEMD_UNIT=podman-homarr.service, org.opencontainers.image.description=Customizable browser's home page to interact with your homeserver's Docker containers (e.g. Sonarr/Radarr), org.opencontainers.image.title=homarr, org.opencontainers.image.url=https://github.com/ajnart/homarr, org.opencontainers.image.version=0.15.2)
Mar 28 21:44:24 host systemd[1]: podman-homarr.service: Main process exited, code=killed, status=9/KILL
Mar 28 21:44:24 host podman[1400301]: 2024-03-28 21:44:24.737847376 +0100 CET m=+10.247879429 container init 532ae295023a3425eb458d43bc24ab5f7d87529a60e3f7fc2872220b4dae0fd0 (image=ghcr.io/ajnart/homarr:latest, name=homarr, org.opencontainers.image.url=https://github.com/ajnart/homarr, org.opencontainers.image.licenses=MIT, org.opencontainers.image.description=Customizable browser's home page to interact with your homeserver's Docker containers (e.g. Sonarr/Radarr), PODMAN_SYSTEMD_UNIT=podman-homarr.service, org.opencontainers.image.title=homarr, org.opencontainers.image.version=0.15.2, org.opencontainers.image.created=2024-03-15T18:49:20.269Z, org.opencontainers.image.revision=5875b1b264c09b362705a1e14cebecf71b12e8ef, org.opencontainers.image.source=https://github.com/ajnart/homarr)
Mar 28 21:44:24 host podman[1400301]: 2024-03-28 21:44:24.740625593 +0100 CET m=+10.250657644 container start 532ae295023a3425eb458d43bc24ab5f7d87529a60e3f7fc2872220b4dae0fd0 (image=ghcr.io/ajnart/homarr:latest, name=homarr, org.opencontainers.image.url=https://github.com/ajnart/homarr, PODMAN_SYSTEMD_UNIT=podman-homarr.service, org.opencontainers.image.description=Customizable browser's home page to interact with your homeserver's Docker containers (e.g. Sonarr/Radarr), org.opencontainers.image.title=homarr, org.opencontainers.image.created=2024-03-15T18:49:20.269Z, org.opencontainers.image.licenses=MIT, org.opencontainers.image.revision=5875b1b264c09b362705a1e14cebecf71b12e8ef, org.opencontainers.image.source=https://github.com/ajnart/homarr, org.opencontainers.image.version=0.15.2)
Mar 28 21:44:24 host homarr[1400521]: Exporting hostname...
Mar 28 21:44:24 host homarr[1400521]: Migrating database...
Mar 28 21:44:24 host homarr[1400521]: yarn run v1.22.19
Mar 28 21:44:25 host homarr[1400521]: $ ts-node ./migrate.ts
Mar 28 21:44:27 host homarr[1400521]: Done in 2.74s.
Mar 28 21:44:27 host homarr[1400521]: Starting production server...

[megaadam]

This is old, but to whoever ends up here...
I have had the same thing. I found that I could add to the service file

[service]
... all other stuff already here
TimeoutStopSec=2s 
# stop waiting, the process does stop, but systemd does not get notified

Yes, it does appear dangerous. It is pretty good if you have bash in your container for debugging. The first thing I saw was if I did watch ps aux in one terminal and stopped the service in another. After systemctl stop the process did die quickly, but system was still hanging. In fact even with kill -15 systemd was hanging. I have not tried ...=0s I leave that for the daring ones!

We are containerizing a large bunch of services, but only a single one misbehaves. I wish I knew why.