btrfs raid not ready but systemd tries to mount it anyway

[danieljrmay]

Describe the bug
I have a machine with a EXT4 / partition and a 10 HDD BTRFS RAID1 volume which I have mounting at /srv in /etc/fstab. Out of the 10 HDDs which make up the BTRFS volume 4 were connected directly to the motherboard, 6 were connected via an HBA card. In this configuration the the BTRFS volume would fail to mount automatically at boot time.

I initially reported this issue at ask.fedoraproject.org:

https://ask.fedoraproject.org/t/btrfs-volume-mount-fails-at-boot-but-works-once-system-is-up/9593

The same issue was then reported to the systemd-devel mailing list:

https://lists.freedesktop.org/archives/systemd-devel/2020-October/045399.html

Where it has been suggested that the issue may be with dracut:

https://lists.freedesktop.org/archives/systemd-devel/2020-October/045440.html

Hence I am reporting this here.

Distribution used
Fedora 32.

Dracut version

$ sudo rpm -q dracut
dracut-050-61.git20200529.fc32.x86_64

Init system

$ sudo rpm -q systemd
systemd-245.8-2.fc32.x86_64

To Reproduce
This may be due to a race condition so it might be difficult to reproduce. However, a setup similar to mine may well have the same problem: non-BTRFS / filesystem, a multi-device BTRFS filesystem which is split between direct connection to the motherboard and HBA card and is configured to mount in /etc/fstab

The BTRFS volume mounts successfully at boot time when:

• udev.log_priority=debug systemd systemd.log_level=debug is added to the grub boot parameters.
• All 10 HDDs are connected to the HBA card (with normal grub boot parameters).

Expected behavior
The BTRFS volume should mount automatically during the boot process.

Additional context
This is a Fedora 32 machine which I think was upgraded via dnf from Fedora 31 sometime ago. The BTRFS volume was added recently when migrating storage from an mdadm based RAID6 XFS filesystem.

I hope this makes some kind of sense 😉

[cmurf]

I agree with Lennart. Either include the btrfs udev rule unconditionally or if fstab contains any btrfs reference. It's definitely not in @danieljrmay 's initramfs when inspected with lsinitrd.

[danieljrmay]

It's definitely not in @danieljrmay 's initramfs when inspected with lsinitrd.

I should have mentioned this in the original report:

$ sudo uname -a
Linux oxygen 5.8.13-200.fc32.x86_64 #1 SMP Thu Oct 1 21:49:42 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ sudo lsinitrd /boot/initramfs-5.8.13-200.fc32.x86_64.img | grep btrfs
$ # No mentions of btrfs found

$ sudo dracut -f

$ sudo lsinitrd /boot/initramfs-5.8.13-200.fc32.x86_64.img | grep btrfs
$ # No mentions of btrfs found

[stale]

This issue is being marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. If this is still an issue in the latest release of Dracut and you would like to keep it open please comment on this issue within the next 7 days. Thank you for your contributions.

[cmurf]

dracut-051 doesn't have a fix for this so I'm commenting to make sure it doesn't get closed.

[stale]

This issue is being marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. If this is still an issue in the latest release of Dracut and you would like to keep it open please comment on this issue within the next 7 days. Thank you for your contributions.

[cmurf]

Buried in the long ask.fpo thread referenced at the top, but not mentioned in this issue, using dracut -f --add btrfs fixes this problem. This results in:

$ sudo lsinitrd /boot/initramfs-5.10.7-200.fc33.x86_64.img | grep btrfs
Arguments: -f --add 'btrfs'
btrfs
-rw-r--r--   1 root     root           20 Oct 23 09:30 etc/cmdline.d/00-btrfs.conf
-rw-r--r--   1 root     root          616 Oct 23 09:30 usr/lib/udev/rules.d/64-btrfs.rules
-rwxr-xr-x   1 root     root      1014296 Oct 23 09:30 usr/sbin/btrfs
lrwxrwxrwx   1 root     root            5 Oct 23 09:30 usr/sbin/btrfsck -> btrfs
-rwxr-xr-x   1 root     root         1189 Oct 23 07:25 usr/sbin/fsck.btrfs
$ 

[haraldh]

But your rootfs != btrfs... Shouldn't it bring up the btrfs after the switch root just fine?
Your initramfs did not even have the btrfs kernel module. So in theory everything should be just working in hotplug mode in the real root. .... Except, if your real root does not wait for the btrfs device to be complete before it tries to mount it to /srv

[haraldh]

To be clear, dracut must not be involved for /srv , if the btrfs devices are totally different from the rootfs device.

That means, initramfs boot must not be stalled for e.g. big SAN arrays mounted to /srv.

[haraldh]

ok, I stand corrected:

Ths btrfs udev rule file appears to be missing in the initrd. The
block devices with the btrfs file systems on them will thus be marked
ready in systemd instantly instead of being delayed until all other
devices of the same btrfs fs have shown up in udev too.

[haraldh]

That is really unfortunate.

[haraldh]

This needs a systemd patch and a dracut patch