createHome attempts to create home directories before local filesystems are mounted

[ToxicFrog]

If a user has createHome = true set -- which is the default for normal users and some services, like gitolite -- whatever process tries to create missing home directories runs after / is mounted but before local-fs.target is reached. This seems to work ok if all mounts are explicit in hardware-configuration.nix (although I haven't tested this thoroughly), but breaks if, e.g., you have ZFS datasets that are mounted by the zfs-mount unit as part of reaching local-fs.target; either services like gitolite-init.service fail because the home directory they expected to exist hasn't been properly created, or zfs-mount.service fails because a bunch of home directories have been created in what it expected to be empty mountpoints.

On my system local-fs.target is reached relatively late (it has a bunch of disks to spin up and zpools to import), so anything that needs local filesystems mounted but doesn't explicitly place itself after local-fs.target tends to go wrong.

This seems like it should be an easy fix (make whatever service creates the home directories after = ["local-fs.target"]), but I can't figure out which service does the deed.

Technical details

• System: NixOS 16.09
• Nix version: 1.11.4
• Nixpkgs version: 16.09.1465.fd1dbe5

[Mic92]

Mounting zfs datasets via zfs native mounting infrastructure is broken in many places on Linux.
For example if one create a bind mount on top of zfs, it is likely the systemd mount target is faster then zfs, when mounting. The bind mount will be present before the underlying mount is ready. Also zfs does not align well with udev. These days we have support for hotplugging, but zfs assumes all hardware is present at boot. I think the only sane solution would be to have a systemd generator, from which systemd can learn what mounts will be created by zfs itself. Until such a generator is written I think we should stick back to zfs's legacy mount feature and specify mountpoints manually.

Having that said: I suspect that zfs is a special case, because otherwise every service would require local-fs.target or we would get way more bug reports in this direction.

Update this is the corresponding upstream bug report: openzfs/zfs#4898

Update here is even a pull request. I will try to review this shortly: openzfs/zfs#4943

[ToxicFrog]

Thanks for the pointers -- that sheds a lot more light on it for me!

It looks like the correct behaviour here is that individual units list the specific mount points that they depend on, and systemd makes sure they don't start until those filesystems are mounted; they don't need to be ordered after local-fs.target, which isn't reached until all local filesystems are mounted. So, e.g., something that depends on /srv but not /home can start up as soon as /srv is mounted even if /home isn't available yet.

ZFS breaks that assumption because systemd doesn't know where the zfs mount points are unless you set mountpoint=legacy and manage the mounts by hand (which I haven't been doing for most mounts because I have 30+ datasets).

This explains why I'm seeing these problems but things work in general even without an after=local-fs.target ordering, and it also explains why making sure that zfs mount -a runs before local-fs.target is reached doesn't actually fix things.

zfs#4943 looks promising as a "proper" fix for this.

In the meantime, making sure that zfs datasets get mounted before sysinit.target seems to be a functioning workaround:

systemd.targets.zfs = {
  wantedBy = ["local-fs.target" "multi-user.target"];
  wants = ["zfs-mount.service"];
  before = ["local-fs.target" "multi-user.target" "sysinit.target"];
};
systemd.services.zfs-mount.requires = ["zfs-import.target"];

[domenkozar]

Duplicates #6481