catatonit keeps a mount directory open, preventing unmounting

[hughsw]

Debian bookworm (lite, headless) on a Raspberry Pi, using podman, although no containers are running.

We have a directory that is a cifs mount of an SMB file system (from a Mac). We could not unmount the directory because it was busy. lsof showed that a catatonit process/thread (related to podman, I believe) was holding the directory open.

I can imagine that catatonit is interested in mounted file systems, but, I would not expect catatonit to be so intrusive as to hold a file/directory open, thus prevent unmounting.

Regards

[cyphar]

Does this happen with catatonit v0.1.6 and later? We added can de to close all fds >=3 a while ago (see #14 and #12) and so it seems unlikely that we would have any fds open as a result. Pre-0.1.6 it was possible for us to keep files open that were passed to the container process, which looks similar to what you're describing.

(FWIW, catatonit is a very simple program, it only opens /dev/tty and possibly /proc/self/fd and closes them before doing a signal passing loop.)

Also, if no containers are running there should be no catatonit process -- catatonit is a pid1 for containers.

[hughsw]

$ catatonit --version
tini version 0.1.7_catatonit

It appears that podman is using the -P option of catatonit as per user.slice -> podman-pause-0db74da4.scope:

$ systemctl status
● raspberrypi3a
    State: running
    Units: 305 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
  systemd: 252.17-1~deb12u1
   CGroup: /
           ├─init.scope
           │ └─1 /sbin/init
           ├─system.slice
             snip
           └─user.slice
             └─user-1000.slice
               ├─session-1.scope
                 snip
               └─user@1000.service
                 ├─init.scope
                 │ ├─714 /lib/systemd/systemd --user
                 │ └─715 "(sd-pam)"
                 ├─session.slice
                 │ └─dbus.service
                 │   └─841 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
                 └─user.slice
                   └─podman-pause-0db74da4.scope
                     └─829 catatonit -P

Catatonit has several objects reported by lsof, where /mnt/wd is the mount that catatonit keeps busy even if no other processes are using it. And that mount is created after the podman pause service is running, suggesting to me that catatonit responds to some notification about the filesystem, and maybe sets its current working directory there...?

$ lsof | fgrep catatonit
COMMAND     PID   TID TASKCMD               USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME
catatonit   829                               pi  cwd       DIR               0,40        0   18989662 /mnt/wd
catatonit   829                               pi  rtd       DIR              179,2     4096          2 /
catatonit   829                               pi  txt       REG              179,2   600176      34185 /usr/bin/catatonit
catatonit   829                               pi    0u      CHR                1,3      0t0          5 /dev/null
catatonit   829                               pi    1u      CHR                1,3      0t0          5 /dev/null
catatonit   829                               pi    2u      CHR                1,3      0t0          5 /dev/null
catatonit   829                               pi    3u  a_inode               0,13        0       1060 [signalfd]

[hughsw]

I said:

... where /mnt/wd is the mount that catatonit keeps busy even if no other processes are using it. And that mount is created after the podman pause service is running...

I'm no longer sure about that claim regarding the order of things.

What I have definitely just seen is that if the podman catatonit -P starts after the cifs mount is already in place, the catatonit process has the cwd entry for the mounted directory. That is, I don't have evidence that catatonit is actively responding to filesystem changes -- I do have evidence that when runs it does something special with the existing cifs mount...

[cyphar]

Ah, the issue is that catatonit is being run inside the mount (by podman presumably). I guess we could change directory to / but if the process is being run with the mount as the root of the mount namespace there's not much we can do. If anything this seems like a podman issue.

What does ls -ld /proc/$(pgrep catatonit)/root output?

[hughsw]

I'll have to check if podman is being used from the mount directory. That will require a reboot and there's a big build ongoing, so it will have to wait until tomorrow...

$ ls -ld /proc/$(pgrep catatonit)/root
lrwxrwxrwx 1 pi pi 0 Nov  4 02:09 /proc/1375/root -> /

[hughsw]

I believe your suspicions are correct. It appears that the first time (in the user-specific systemd slice) that a user runs podman it creates a "hidden" container running catatonit -P with cwd being the directory from which podman was run. Subsequent use of podman from other directories (or other simultaneous logins by the user) has no effect on the hidden container. This container does not show up in podman ps, and it persists until the user-specific slice terminates.

I will file an issue with the podman project. It seems to me that the hidden container should have its cwd in the user's home directory, or some suitable /var area.

[mihalyr]

I just ran into the same thing on Fedora 40, I couldn't unmount a target and catatonit -P was the culprit. I used toolbox container on the mount, but the catatonit process still holds the mount even after I killed the toolbox container.

I will file an issue with the podman project. It seems to me that the hidden container should have its cwd in the user's home directory, or some suitable /var area.

Is there an issue on podman side that we can track?

[cyphar]

Does #33 fix the issue?

[cyphar]

catatonit 0.2.1 includes the fix for this.