OpenZFS not mounting on reboot anymore

[lastb0isct]

Type	Version/Name
Distribution Name	RHEL8
Distribution Version	8_6
Kernel Version	4.18.0-372.26.1.
Architecture	. x86_64
OpenZFS Version	zfs-2.1.6-1 & zfs-kmod-2.1.6-1

Describe the problem you're observing

ZFS Volumes do not mount on boot. I think that there is an issue with zfs-snapshot-bootfs.service. It does not exist on my system after upgrading to 2.1.6. My dmesg shows this:

dmesg | grep zfs
[    1.581956] systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:11: Unknown lvalue '-ExecStart' in section 'Service'
[    1.582174] systemd[1]: zfs-snapshot-bootfs.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
[    1.591136] systemd[1]: zfs-snapshot-bootfs.service: Cannot add dependency job, ignoring: Unit zfs-snapshot-bootfs.service has a ba
d unit file setting.

I have to then run systemctl restart zfs-import-scan.service & systemctl restart zfs-mount.service to mount the pools.

Describe how to reproduce the problem

Just reboot and voila nothing is mounted.

Include any warning/errors/backtraces from the system logs

This is the /var/log/messages | grep zfs:

Feb  5 17:21:05 PLEX systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:11: Unknown lvalue '-ExecStart' in section 'Servi
ce'
Feb  5 17:21:05 PLEX systemd[1]: zfs-snapshot-bootfs.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
Feb  5 17:21:05 PLEX systemd[1]: zfs-snapshot-bootfs.service: Cannot add dependency job, ignoring: Unit zfs-snapshot-bootfs.service ha
s a bad unit file setting.
Feb  5 17:21:05 PLEX systemd-modules-load[585]: Inserted module 'zfs'
Feb  5 17:21:06 PLEX systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:11: Unknown lvalue '-ExecStart' in section 'Servi
ce'
Feb  5 17:21:06 PLEX systemd[1]: zfs-snapshot-bootfs.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
Feb  5 17:21:09 PLEX systemd[1201]: zfs-import-scan.service: Failed to execute command: No such file or directory
Feb  5 17:21:09 PLEX systemd[1201]: zfs-import-scan.service: Failed at step EXEC spawning /usr/local/sbin/zpool: No such file or direc
tory
Feb  5 17:21:09 PLEX systemd[1]: zfs-import-scan.service: Main process exited, code=exited, status=203/EXEC
Feb  5 17:21:09 PLEX systemd[1]: zfs-import-scan.service: Failed with result 'exit-code'.
Feb  5 17:21:09 PLEX systemd[1]: zfs-env-bootfs.service: Succeeded.
Feb  5 17:21:09 PLEX systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:11: Unknown lvalue '-ExecStart' in section 'Servi
ce'
Feb  5 17:21:09 PLEX systemd[1]: zfs-snapshot-bootfs.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

[rincebrain]

Looks like fallout from 979fd5a if I had to guess.

@gregory-lee-bartholomew I would guess your change was incorrect, as it was "ExecStart=-..." before 8213ddf and your change made it "-ExecStart=...".

[gregory-lee-bartholomew]

It wasn't my doing. Someone else mangled the commit.

[lastb0isct]

The issue is that zfs-snapshot-bootfs.service doesn't even exist! It is removed so the service itself can't start...because there is not .service file.

[lastb0isct]

systemctl cat zfs-snapshot-bootfs.service
No files found for zfs-snapshot-bootfs.service.

[gregory-lee-bartholomew]

It won't be in the systemd libraries of the running system. Dracut injects it into the initramfs from /usr/lib/dracut/modules.d/90zfs.

[gregory-lee-bartholomew]

As a quick fix until you can upgrade to a newer version of zfs-dracut that is fixed, you can edit the /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service file directly and regenerate your initramfs.

[gregory-lee-bartholomew]

This is what mine currently looks like.

# rpm -qf /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service
zfs-dracut-2.1.7-1.fc36.noarch

# cat /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service
[Unit]
Description=Snapshot bootfs just before it is mounted
Requisite=zfs-import.target
After=zfs-import.target dracut-pre-mount.service
Before=dracut-mount.service
DefaultDependencies=no
ConditionKernelCommandLine=bootfs.snapshot

[Service]
Type=oneshot
ExecStart=-/bin/sh -c '. /lib/dracut-zfs-lib.sh; decode_root_args || exit; [ "$root" = "zfs:AUTO" ] && root="$BOOTFS"; SNAPNAME="$(getarg bootfs.snapshot)"; exec /sbin/zfs snapshot "$root@${SNAPNAME:-%v}"'
RemainAfterExit=yes

[rincebrain]

Interesting. Mismerge, then? Surprising nobody got burned by it sooner...

I'll try to remember to open a trivial fix PR for 2.1.10 later if nobody beats me to it.

[gregory-lee-bartholomew]

Yeah, for the record: #13769 (comment)

[gregory-lee-bartholomew]

I'll try to remember to open a trivial fix PR for 2.1.10 later if nobody beats me to it.

I think it is already fixed in more recent versions.

[rincebrain]

Huh, I was looking at 2.1-release and saw it look like that I thought...nope, you're right, it's fixed in 2.1.7.

Thanks! I hadn't seen that kerfuffle with 2.1.6.

[lastb0isct]

Interesting that rpm -qf /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service outputs this:

file /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service is not owned by any package

[lastb0isct]

I will try and reboot the system tomorrow when i'm back in front of it. Thank you for the guidance!

[gregory-lee-bartholomew]

Sounds like a corrupt rpm database. Maybe rpmdb --rebuilddb would fix it?

[lastb0isct]

Doesn't resolve it. Shouldn't matter though, should it?

rpmdb --rebuilddb
rpm -qf /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service
file /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service is not owned by any package

[gregory-lee-bartholomew]

If the files under /var/lib/rpm are missing or corrupt, then rpm -qf may not work. Does rpm -ql zfs-dracut show anything?

# rpm -ql zfs-dracut
/usr/lib/dracut/modules.d/02zfsexpandknowledge
/usr/lib/dracut/modules.d/02zfsexpandknowledge/module-setup.sh
/usr/lib/dracut/modules.d/90zfs
/usr/lib/dracut/modules.d/90zfs/export-zfs.sh
/usr/lib/dracut/modules.d/90zfs/import-opts-generator.sh
/usr/lib/dracut/modules.d/90zfs/module-setup.sh
/usr/lib/dracut/modules.d/90zfs/mount-zfs.sh
/usr/lib/dracut/modules.d/90zfs/parse-zfs.sh
/usr/lib/dracut/modules.d/90zfs/zfs-env-bootfs.service
/usr/lib/dracut/modules.d/90zfs/zfs-generator.sh
/usr/lib/dracut/modules.d/90zfs/zfs-lib.sh
/usr/lib/dracut/modules.d/90zfs/zfs-load-key.sh
/usr/lib/dracut/modules.d/90zfs/zfs-needshutdown.sh
/usr/lib/dracut/modules.d/90zfs/zfs-rollback-bootfs.service
/usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service
/usr/share/doc/zfs-dracut
/usr/share/doc/zfs-dracut/README.dracut.markdown

[lastb0isct]

Yep, but not everything it seems.

rpm -ql zfs-dracut
/usr/lib/dracut/modules.d/02zfsexpandknowledge
/usr/lib/dracut/modules.d/02zfsexpandknowledge/module-setup.sh
/usr/lib/dracut/modules.d/90zfs
/usr/lib/dracut/modules.d/90zfs/export-zfs.sh
/usr/lib/dracut/modules.d/90zfs/module-setup.sh
/usr/lib/dracut/modules.d/90zfs/mount-zfs.sh
/usr/lib/dracut/modules.d/90zfs/parse-zfs.sh
/usr/lib/dracut/modules.d/90zfs/zfs-env-bootfs.service
/usr/lib/dracut/modules.d/90zfs/zfs-generator.sh
/usr/lib/dracut/modules.d/90zfs/zfs-lib.sh
/usr/lib/dracut/modules.d/90zfs/zfs-load-key.sh
/usr/lib/dracut/modules.d/90zfs/zfs-needshutdown.sh
/usr/share/doc/zfs-dracut
/usr/share/doc/zfs-dracut/README.dracut.markdown

[gregory-lee-bartholomew]

Very odd. Sorry, I don't know what is going on there.

[rincebrain]

If Plex is modifying the packages, maybe it's a question for them?

It seems kind of academic, but that would be what I'd do - go compare their specfile with the stock one.

[lastb0isct]

Plex is not doing anything...that is the hostname of my server, lol

[rincebrain]

Oh, I see. In your original paste, it had rpm -ql zfs-dracutPLEX, which confused me. I see you've edited it.

[lastb0isct]

Going to open a new ticket, but trying to update zfs it seems I have both dkms & kmod on my system...will ask what the best way forward is for that in a separate ticket.

[lastb0isct]

Updated today -- will restart when time permits.