dracut: zfs_force=1 has no effect

[rkitover]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	unstable profile default/linux/amd64/17.1/desktop/plasma/systemd
Linux Kernel	5.9
Architecture	amd64
ZFS Version	git master
SPL Version	git master

Describe the problem you're observing

When my zfs pool needs to be force-imported because I booted freebsd on it, the kernel commandline option I added zfs_force=1 does not have any effect, and I have to do a zpool import -f from the dracut rescue shell.

Describe how to reproduce the problem

Add zfs_force=1 to GRUB_CMDLINE_LINUX in /etc/default/grub and regen grub config.

Do something that would require force-importing the pool, in my case that is booting FreeBSD on it.

When you boot linux, you will see dracut fail to import the pool and drop you into the emergency shell.

Include any warning/errors/backtraces from the system logs

[nabijaczleweli]

Could you try the top commit off my suseł-but-stronger branch (referenced above)? It worked in my testing, but, y'know

[rkitover]

Hi, thank you for your response.

I am away from my desktop system right now where I use dracut, on my laptop I have NixOS which I believe does not use dracut.

If I don't get back to my desktop in the next few days to try this, I will just install another distribution on the pool on my laptop, where I have FreeBSD as well already.

How's installing Ubuntu on an existing pool these days? I saw they were doing some serious work in that area. Does Ubuntu use dracut? I'll try it if that's the case.

Your patch looks great by the way.

[bghira]

it works here on Ubuntu, though getting dracut to work on Ubuntu in general is a bit of a gamble.

[rkitover]

So I just got back home recently and tried this on my desktop, with freebsd and gentoo, dracut still drops me to the emergency shell. When I was in the emergency shell, I verified that the patch was applied to my initramfs.

I'm happy to do any debugging.

[nabijaczleweli]

Hm, so if you break in the initrd (or fall through to the emergency shell), these should all be positive:

For the generator:

1. do you definitely have /lib/systemd/system-environment-generators/zfs-import-opts.sh (and it's executable)?
2. does systemctl --system show-environment list ZPOOL_IMPORT_OPTS= at all? (should be -f when you set zfs_force=1)?
3. does it appear if you do systemctl daemon-reload?
4. what do you get on stdout if you run it directly – is it just the ZPOOL_IMPORT_OPTS line?
5. if you append something like set > /dev/console (or /dev/kmsg, or whichever tty you get the shell at) to the generator and reload again, do you get any output (if not – what can you do to get it to run)?

And for the services:

1. do the /lib/systemd/system/zfs-import-{cache,scan}.d/import-opts.conf drop-ins exist? (or maybe they're in the initrd, but in the wrong place? it's entirely possible I got the path wrong!)
2. does systemctl cat zfs-import* list both the service(s) and the drop-ins?
3. if you do systemctl status zfs-import-[the import unit that ran].service – does it list the (value of) $ZPOOL_IMPORT_OPTS in the failed cmdline?

I suspect I just got the permissions/paths/overrides wrong somewhere

[rkitover]

The problem seems to be with my version of systemd not reading the output of the environment generator correctly.

When I do a systemctl daemon-reload I see the WARNING: ZFS will force import pools... etc., and if do a . /lib/dracut-zfs-lib.sh at the prompt then the ZPOOL_IMPORT_OPTS=-f is set in my environment, however systemctl show-environment does not show the variable. I tried removing the leading space before -f but that didn't help.

I did notice that if I do a systemctl set-environment ZPOOL_IMPORT_OPTS=-f then this shows up in systemctl show-environment.

[nabijaczleweli]

Environment generators seem to've landed in systemd 233 (2017-03-01) and remained unchanged since, and the versions I see on packages.g.o are 9999(?) and 248.3, both of which are newer than the 241 from buster that I test on.

I assume the warning you get is from the alternate output, since stderr is devnulled (unless it isn't maybe? but it was in my tests); what happens if you add echo "a b=c" just after the shebang? You should get a warning like

May 30 20:44:32 tarta systemd[8346]: Invalid variable assignment "a b=...", ignoring.

in the journal. If you don't – can you try moving it to, like, /etc/systemd/system-environment-generators maybe?

It might be necessary to fall back to a normal service if your systemd really is so terminally broken so as to have completely non-functional environment generators but I somehow doubt this, given that they've not changed (save for the umask they get) since 4 years ago.

[rkitover]

This system is a couple of months old and my version of systemd is 247.2-r4.

I am going to rebuild my initramfs with vi or something and play with this some more.

[rkitover]

Good news, I found the problem.

It seems that /lib/systemd/system-environment-generators is ignored on my system and the warning I saw was from another invocation of that script.

This may be because this version of systemd only looks in /usr/lib I don't know.

But once I copied the zfs-imports.sh file to /etc/systemd/system-environment-generators, reran daemon-reload and show-environment the variable was set.

[nabijaczleweli]

Hm, odd. What's pkg-config --variable=systemdsystemunitdir systemd (and systemdutildir/systemdsystemgeneratordir) for you? The script installs the units there and the generator (parallel to) there as well, so the script should land in /u/l/s/s-e-g/ in your initrd.

[rkitover]

Here's what I see on my system, environment generators seem to be in /usr/lib:

✔  ~
rkitover@epyc  ➤  ls /usr/lib/systemd/system-environment-generators/
10-gentoo-path  60-flatpak-system-only

While the variables point to /lib, some of them:

✔  ~
rkitover@epyc  ➤  pkg-config --variable=systemdsystemunitdir systemd
/lib/systemd/system
✔  ~
rkitover@epyc  ➤  pkg-config --variable=systemdutildir systemd
/lib/systemd
✔  ~
rkitover@epyc  ➤  pkg-config --variable=systemdsystemgeneratordir systemd
/lib/systemd/system-generators

Here are all the variables:

✔  ~
rkitover@epyc  ➤  pkg-config --print-variables systemd | while read -r v; do printf "%-35s%s\n" "$v" "$(pkg-config --variable=$v systemd)"; done
containeruidbasemax                1878982656
container_uid_base_max             1878982656
containeruidbasemin                524288
container_uid_base_min             524288
dynamicuidmax                      65519
dynamic_uid_max                    65519
dynamicuidmin                      61184
dynamic_uid_min                    61184
systemgidmax                       999
system_gid_max                     999
systemuidmax                       999
system_uid_max                     999
catalogdir                         /usr/lib/systemd/catalog
catalog_dir                        /usr/lib/systemd/catalog
modulesloaddir                     /lib/modules-load.d
modules_load_dir                   /lib/modules-load.d
binfmtdir                          /lib/binfmt.d
binfmt_dir                         /lib/binfmt.d
sysctldir                          /lib/sysctl.d
sysctl_dir                         /lib/sysctl.d
sysusersdir                        /lib/sysusers.d
sysusers_dir                       /lib/sysusers.d
tmpfilesdir                        /usr/lib/tmpfiles.d
tmpfiles_dir                       /usr/lib/tmpfiles.d
systemdshutdowndir                 /lib/systemd/system-shutdown
systemd_shutdown_dir               /lib/systemd/system-shutdown
systemdsleepdir                    /lib/systemd/system-sleep
systemd_sleep_dir                  /lib/systemd/system-sleep
systemdusergeneratorpath           /run/systemd/user-generators:/etc/systemd/user-generators:/usr/local/lib/systemd/user-generators:/usr/lib/systemd/user-generators
systemd_user_generator_path        /run/systemd/user-generators:/etc/systemd/user-generators:/usr/local/lib/systemd/user-generators:/usr/lib/systemd/user-generators
systemdsystemgeneratorpath         /run/systemd/system-generators:/etc/systemd/system-generators:/usr/local/lib/systemd/system-generators:/lib/systemd/system-generators
systemd_system_generator_path      /run/systemd/system-generators:/etc/systemd/system-generators:/usr/local/lib/systemd/system-generators:/lib/systemd/system-generators
systemdusergeneratordir            /usr/lib/systemd/user-generators
systemd_user_generator_dir         /usr/lib/systemd/user-generators
systemdsystemgeneratordir          /lib/systemd/system-generators
systemd_system_generator_dir       /lib/systemd/system-generators
systemduserunitpath                /etc/systemd/user:/etc/systemd/user:/run/systemd/user:/usr/local/lib/systemd/user:/usr/local/share/systemd/user:/usr/lib/systemd/user:/usr/lib/systemd/user:/usr/share/systemd/user
systemd_user_unit_path             /etc/systemd/user:/etc/systemd/user:/run/systemd/user:/usr/local/lib/systemd/user:/usr/local/share/systemd/user:/usr/lib/systemd/user:/usr/lib/systemd/user:/usr/share/systemd/user
systemdsystemunitpath              /etc/systemd/system:/etc/systemd/system:/run/systemd/system:/usr/local/lib/systemd/system:/lib/systemd/system:/usr/lib/systemd/system:/lib/systemd/system
systemd_system_unit_path           /etc/systemd/system:/etc/systemd/system:/run/systemd/system:/usr/local/lib/systemd/system:/lib/systemd/system:/usr/lib/systemd/system:/lib/systemd/system
systemduserconfdir                 /etc/systemd/user
systemd_user_conf_dir              /etc/systemd/user
systemdsystemconfdir               /etc/systemd/system
systemd_system_conf_dir            /etc/systemd/system
systemduserpresetdir               /usr/lib/systemd/user-preset
systemd_user_preset_dir            /usr/lib/systemd/user-preset
systemduserunitdir                 /usr/lib/systemd/user
systemd_user_unit_dir              /usr/lib/systemd/user
systemdsystempresetdir             /lib/systemd/system-preset
systemd_system_preset_dir          /lib/systemd/system-preset
systemdsystemunitdir               /lib/systemd/system
systemd_system_unit_dir            /lib/systemd/system
systemdutildir                     /lib/systemd
systemd_util_dir                   /lib/systemd
sysconfdir                         /etc
sysconf_dir                        /etc
rootprefix                         
root_prefix                        
prefix                             /usr
pcfiledir                          /usr/share/pkgconfig

Gentoo says the following about the ebuild I have installed:

sys-apps/systemd-247.2-r4::gentoo was built with the following:
USE="acl -apparmor audit -build cgroup-hybrid cryptsetup curl dns-over-tls elfutils gcrypt gnuefi homed http hwdb idn importd kmod lz4 lzma nat pam pcre pkcs11 (policykit) pwquality qrcode repart resolvconf seccomp (-selinux) (split-usr) -static-libs sysv-utils -test -vanilla xkb zstd" ABI_X86="32 (64) (-x32)"
FEATURES="distlocks strict-keepdir unmerge-orphans binpkg-dostrip preserve-libs parallel-fetch ccache split-elog buildpkg ebuild-locks news userfetch config-protect-if-modified fixlafiles qa-unresolved-soname-deps unknown-features-warn sandbox unmerge-logs multilib-strict sfperms strict assume-digests binpkg-logs merge-sync usersync protect-owned ipc-sandbox binpkg-docompress split-log pid-sandbox xattr"

[nabijaczleweli]

That without doubt is one of the worst things I've ever read!

Turns out the environment generator path is

add = strv_new("/run/systemd/system-environment-generators",
                "/etc/systemd/system-environment-generators",
                "/usr/local/lib/systemd/system-environment-generators",
                SYSTEM_ENV_GENERATOR_DIR);

and

systemenvgeneratordir = join_paths(prefixdir, 'lib/systemd/system-environment-generators')
systemunitdir = join_paths(rootprefixdir, 'lib/systemd/system')

so it only works by accident on merged-/usr systems.

Can you try the top of my suseł-but-stronger-v2 branch (referenced above)? lsinitrd | grep zfs-import-opts.sh should yield /usr/lib/... instead of /lib/... now, and it should Just Work

[rkitover]

Yup that works!

Thank you very much for fixing this by the way, this will make it much easier for me to dual-boot Linux and FreeBSD, once I reinstall it completely, there's something screwy in this install and no DE works correctly. But now I can boot with amdgpu throwing a kernel panic only 20% of the time.