jectl umount fails on jail restart

[Subito]

I have a strange new problem: restarting jails does not work after I activated a new jailenv. It seems that the parent dataset can't be unmounted. The strange thing is: After "a while" it just works again, without me messing with jectl or zfs (all Jails are stopped at this moment):

root@hz28:~ # jectl umount ngxtest1
cannot unmount '/usr/jails/ngxtest1': pool or dataset is busy
root@hz28:~ # jectl
usage: jectl <command> ...
[...] 
root@hz28:~ # jectl list
NAME                                                USED  AVAIL     REFER  MOUNTPOINT
zroot/JAIL                                         2.23G   437G       96K  none
zroot/JAIL/miniotest                               1.12G   437G       96K  none
zroot/JAIL/miniotest/2022-08-09_1037_minio         1.41M   437G     1.12G  /usr/jails/miniotest
zroot/JAIL/miniotest/2022-08-09_1037_minio/config   712K   437G      612K  /usr/jails/miniotest/config
zroot/JAIL/miniotest/2022-08-09_1037_minio/minio     96K   437G       96K  /usr/jails/miniotest/minio
zroot/JAIL/miniotest/default                       1.12G   437G     1.12G  /usr/jails/miniotest
zroot/JAIL/ngxtest1                                1.12G   437G       96K  none
zroot/JAIL/ngxtest1/2022-08-09_1037_nginx          1.46M   437G     1.11G  /usr/jails/ngxtest1
zroot/JAIL/ngxtest1/2022-08-09_1037_nginx/config    780K   437G      672K  /usr/jails/ngxtest1/config
zroot/JAIL/ngxtest1/2022-08-09_1037_nginx/sites      96K   437G       96K  /usr/jails/ngxtest1/sites
zroot/JAIL/ngxtest1/default                        1.11G   437G     1.11G  /usr/jails/ngxtest1
zroot/JE                                           4.46G   437G       96K  none
zroot/JE/2022-08-08_1431_minio                     1.12G   437G     1.12G  none
zroot/JE/2022-08-08_1431_nginx                     1.11G   437G     1.11G  none
zroot/JE/2022-08-09_1037_minio                     1.12G   437G     1.12G  none
zroot/JE/2022-08-09_1037_nginx                     1.11G   437G     1.11G  none
root@hz28:~ # jectl umount ngxtest1
root@hz28:~ # 

Any ideas on how to debug this further?

[Subito]

It seems this also affects normal restarts. Most times it fails with "pool or dataset busy" and starts working again "after a bit". This is not ideal, as it makes restarting jails very hit-or-miss. Do you have any guesses what might be the problem here?

[rob-wing]

I'll look into this.

When you encounter the 'cannot mount' error, grab the output of jls -v.

Could you provide a quick rundown of the steps that allow you to reproduce this problem?

Also, what's the output of uname -a?

[Subito]

Following steps to reproduce:

jail.conf.d/miniotest.conf:

path = "/usr/jails/$name";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
#exec.prepare = "/usr/local/sbin/jupdate.sh -u $name";
exec.prepare += "jectl mount $name $path";

miniotest {
   host.hostname = miniotest;
   ip6.addr = "dead:beef::";
   ip4.addr = 127.0.0.10;
   exec.prestart+="logger starting $name";
}

Then start the jail and try restarting:

root@hz28:~ # service jail restart miniotest
Stopping jails: miniotest.
Starting jails: cannot start jail  "miniotest":
cannot unmount '/usr/jails/miniotest': pool or dataset is busy
jail: miniotest: jectl mount miniotest /usr/jails/miniotest: failed

jls -v just after trying to restart:

root@hz28:~ # jls -v
   JID  Hostname                      Path
        Name                          State
        CPUSetID
        IP Address(es)
    17  phptest1.kommune.net          /usr/jails/phptest1
        phptest1                      ACTIVE
        6
    18  miniomirror1.kommune.net      /usr/jails/miniomirror1
        miniomirror1                  ACTIVE
        8
        127.0.0.11
    19  miniomirror2.kommune.net      /usr/jails/miniomirror2
        miniomirror2                  ACTIVE
        10
        127.0.0.12

uname -a: FreeBSD hz28.kommune.net 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64

I basically created a few jails, added a few environments and tried restarting. It "feels" worse, the more jails and environments I add, with just one jail and 1-2 Envs it doesn't seem to happen nearly as often. After waiting for a few minutes, I can start the affected jail again:

root@hz28:~ # service jail restart miniotest
Stopping jails:.
Starting jails: miniotest.

[rob-wing]

It appears there's a hanging reference to the mountpoint which doesn't allow it to unmount immediately.

Can you grab the output of the following commands when the error occurs:

# fstat -v
# sockstat -v
# netstat -an | grep tcp

[Subito]

Of course:
https://gist.github.com/Subito/8852e6f2011200331432ecb4110e185c

anything I could grep for to make the output a bit more readable?

[rob-wing]

I didn't see anything that stood out, unless the jail that failed to unmount before the commands were run was different than the miniotest jail.

My original thought was that when the jail was stopped, a service running in the jail was keeping a socket in the TIME_WAIT state, preventing the unmount until the connection was cleaned up. In the output, I did see a few sockets in the TIME_WAIT state but, it didn't appear to match the IP address or be associated with the miniotest jail.

The other thing I was looking for was to see if any files were open that reside in the jail directory but, I didn't see that either.

Again, when you encounter the error, grab output of:

# jls -na

[Subito]

Of course:

# service jail restart miniotest
Stopping jails:.
Starting jails: miniotest.
# service jail restart miniotest
Stopping jails: miniotest.
Starting jails: cannot start jail  "miniotest":
cannot unmount '/usr/jails/miniotest': pool or dataset is busy
jail: miniotest: jectl mount miniotest /usr/jails/miniotest: failed
.
# jls -na
devfs_ruleset=0 nodying enforce_statfs=2 host=new ip4=disable ip6=disable jid=17 name=phptest1 osreldate=1301000 osrelease=13.1-RELEASE parent=0 path=/usr/jails/phptest1 nopersist securelevel=-1 sysvmsg=disable sysvsem=disable sysvshm=disable vnet=inherit allow.nochflags allow.nomlock allow.nomount allow.mount.nodevfs allow.mount.noprocfs allow.mount.notmpfs allow.mount.nozfs allow.noquotas allow.noraw_sockets allow.noread_msgbuf allow.reserved_ports allow.set_hostname allow.nosocket_af allow.suser allow.nosysvipc allow.unprivileged_proc_debug children.cur=0 children.max=0 cpuset.id=6 host.domainname="" host.hostid=0 host.hostname=phptest1.kommune.net host.hostuuid=00000000-0000-0000-0000-000000000000 ip4.addr= ip4.saddrsel ip6.addr=2a01:4f9:5a:1a8f::12 ip6.saddrsel
devfs_ruleset=0 nodying enforce_statfs=2 host=new ip4=disable ip6=disable jid=18 name=miniomirror1 osreldate=1301000 osrelease=13.1-RELEASE parent=0 path=/usr/jails/miniomirror1 nopersist securelevel=-1 sysvmsg=disable sysvsem=disable sysvshm=disable vnet=inherit allow.nochflags allow.nomlock allow.nomount allow.mount.nodevfs allow.mount.noprocfs allow.mount.notmpfs allow.mount.nozfs allow.noquotas allow.noraw_sockets allow.noread_msgbuf allow.reserved_ports allow.set_hostname allow.nosocket_af allow.suser allow.nosysvipc allow.unprivileged_proc_debug children.cur=0 children.max=0 cpuset.id=8 host.domainname="" host.hostid=0 host.hostname=miniomirror1.kommune.net host.hostuuid=00000000-0000-0000-0000-000000000000 ip4.addr=127.0.0.11 ip4.saddrsel ip6.addr=2a01:4f9:5a:1a8f::13 ip6.saddrsel
devfs_ruleset=0 nodying enforce_statfs=2 host=new ip4=disable ip6=disable jid=19 name=miniomirror2 osreldate=1301000 osrelease=13.1-RELEASE parent=0 path=/usr/jails/miniomirror2 nopersist securelevel=-1 sysvmsg=disable sysvsem=disable sysvshm=disable vnet=inherit allow.nochflags allow.nomlock allow.nomount allow.mount.nodevfs allow.mount.noprocfs allow.mount.notmpfs allow.mount.nozfs allow.noquotas allow.noraw_sockets allow.noread_msgbuf allow.reserved_ports allow.set_hostname allow.nosocket_af allow.suser allow.nosysvipc allow.unprivileged_proc_debug children.cur=0 children.max=0 cpuset.id=10 host.domainname="" host.hostid=0 host.hostname=miniomirror2.kommune.net host.hostuuid=00000000-0000-0000-0000-000000000000 ip4.addr=127.0.0.12 ip4.saddrsel ip6.addr=2a01:4f9:5a:1a8f::14 ip6.saddrsel
devfs_ruleset=0 dying enforce_statfs=2 host=new ip4=disable ip6=disable jid=22 name=miniotest osreldate=1301000 osrelease=13.1-RELEASE parent=0 path=/usr/jails/miniotest nopersist securelevel=-1 sysvmsg=disable sysvsem=disable sysvshm=disable vnet=inherit allow.nochflags allow.nomlock allow.nomount allow.mount.nodevfs allow.mount.noprocfs allow.mount.notmpfs allow.mount.nozfs allow.noquotas allow.noraw_sockets allow.noread_msgbuf allow.reserved_ports allow.set_hostname allow.nosocket_af allow.suser allow.nosysvipc allow.unprivileged_proc_debug children.cur=0 children.max=0 cpuset.id=3 host.domainname="" host.hostid=0 host.hostname=miniotest.kommune.net host.hostuuid=00000000-0000-0000-0000-000000000000 ip4.addr=127.0.0.10 ip4.saddrsel ip6.addr=2a01:4f9:5a:1a8f::11 ip6.saddrsel

somehow the jail is still in the dying phase?

[rob-wing]

Yes it is.

Looking at the output of sockstat -v and netstat -an | grep tcp that you posted earlier, there are connections in the TIME_WAIT state associated with the miniotest jail - this is likely the issue that is preventing the unmount immediately. These connections in the TIME_WAIT state get cleaned up after a couple minutes, which is why you're able to unmount the jail properly after some amount of time passes.

I'm thinking of a clean way to handle this. It is possible to force the unmount but, I'd like to find an alternate solution.

[rob-wing]

I've added a work-around for this problem in commit f6c0032

Let me know if this problem still persists after updating.

[Subito]

That makes it recover faster from the error, and fixes it for simple jail-restarts, but unfortunately it does not fix the problem with activating a new JE and instantly starting the jail afterwards:

root@hz28:~ # service jail stop miniotest && jectl activate miniotest 2023-01-09_0951_minio && service jail start miniotest
Stopping jails: miniotest.
cannot unmount '/usr/jails/miniotest': pool or dataset is busy
root@hz28:~ # jls -na
devfs_ruleset=0 dying enforce_statfs=2 host=new ip4=disable ip6=disable jid=67 linux=new name=miniotest [...]

I assume je_swapin does not force the unmount? Which I think is "right", because it should't always force the unmount, but in this case I can't pass an option or anything to activate to do that, I think?

[rob-wing]

Indeed je_swapin() doesn't force the unmount, hmm...and, I agree, that's probably the "right" thing since you probably don't want to force the unmount if the jail is running.

Perhaps it'd make sense for jectl to force the unmount if the jail is in the dying state..?

A way to workaround this for testing at the moment would be to change your command to something like:

service jail stop miniotest && \
jectl umount -f miniotest &&  \
jectl activate miniotest 2023-01-09_0951_minio && \
service jail start miniotest

or you could perform the unmount using the exec.release hook in jail.conf:

exec.release="jectl umount -f $name"
miniotest {
    ip4.addr = 192.168.1.1;
}

[Subito]

Coming back to this after a while: Is it safe to force the unmount in exec.release? Especially with ZFS I thought it might be possible that writing to disk might be a bit delayed. If we force the unmount every time we restart the jail, wouldn't there be a possibility of data-loss?

[xbeaudouin]

Hello,
Well it's depends of the situation.

• On ZFS point of view, excepted if we are in kernel parnic situation or whatever, we can safely says there will no loss of data
• On the FreeBSD point of view, to be more precise on the Application running in the jail, if it has not finished to write data yet, the next write after forcing unmount will get EBADF (invalid file descriptor) after the unmount. So it is possible you lost data in this case.

We are still working with upstream to resolve the root cause, so that jails will not have to wait for the network to wind down before it can be unmounted.
Kind regards

[Subito]

Hi!

Are there any news to this? I don't think I'll run the risk of force-unmounting on a regular basis on production jails. Is the situation any different or improved on 14.1?

Cheers