initrd-sysroot-mount.service doesn't mount /sysroot

[kubax]

Hi,

i got some problem with your workaround for the systemd update.

My initrd-sysroot-mount doesn't mount the /sysroot and throws an error.

Mär 19 12:18:41 archlinux kernel: BTRFS error (device dm-5): failed to read chunk tree: -22
Mär 19 12:18:41 archlinux kernel: BTRFS error (device dm-5): open_ctree failed
Mär 19 12:18:41 archlinux mount[422]: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/mapper/archroot-ROOT, missing codepage or helper program, or other err>
Mär 19 12:18:41 archlinux systemd[1]: initrd-sysroot-mount.service: Main process exited, code=exited, status=32/n/a
Mär 19 12:18:41 archlinux systemd[1]: initrd-sysroot-mount.service: Failed with result 'exit-code'.
Mär 19 12:18:41 archlinux systemd[1]: Failed to start Initrd SysRoot Mount.

Here is my /etc/mkinitcpio.d/fstab/crypttab

crypttab

#
# Each mapped device will be created in /dev/mapper, so your /etc/fstab
# should use the /dev/mapper/<name> paths for encrypted devices.
#
# The Arch specific syntax has been deprecated, see crypttab(5) for the
# new supported syntax.
#
# NOTE: Do not list your root (/) partition here, it must be set up
#       beforehand by the initramfs (/etc/mkinitcpio.conf).

# <name>       <device>                                     <password>              <options>
# home         UUID=b8ad5c18-f445-495d-9095-c9ec4f9d2f37    /etc/mypassword1
# data1        /dev/sda3                                    /etc/mypassword2
# data2        /dev/sda5                                    /etc/cryptfs.key
# swap         /dev/sdx4                                    /dev/urandom            swap,cipher=aes-cbc-essiv:sha256,size=256
# vol          /dev/sdb7                                    none
cryptlvm       UUID=c0bbc682-770d-4dff-a083-46557c2ef042    none
cryptdatastore UUID=99997e7e-e9dd-4f15-979e-57e9cb2ccbd0    none                    luks

fstab:

# fstab: mappings for direct partitions

# fstab format:
# https://wiki.archlinux.org/index.php/Fstab

# how fstab is used by systemd
# https://www.freedesktop.org/software/systemd/man/systemd-fstab-generator.html
# https://github.com/systemd/systemd/blob/master/src/fstab-generator/fstab-generator.c

# note: provide /sysroot/ folder inside initramfs disk image
# note: remove "root=/dev/mapper/root" stanza from kernel command line

# provide here root partition description (instead of kernel command line)

#  <file system>        <dir>      <type>    <option>                         <dump> <pass>
#  /dev/mapper/root     /sysroot    auto     x-systemd.device-timeout=9999h     0     1
/dev/mapper/archroot-ROOT     /sysroot    btrfs     x-systemd.device-timeout=30m     0     1

Interestingly... When i manually start the service file in the emergency console, it mounts, and can boot.

any hints? :/

[Andrei-Pozolotin]

1. try work around for open_ctree failed:
   https://wiki.archlinux.org/index.php/Btrfs#BTRFS:_open_ctree_failed

2. ensure proper service dependencies in:
   https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/initrd-sysroot-mount.service#L11

[kubax]

Thanks for the Answer.

Sadly both didn't work / is allready applied.

1. i'm allready using systemd instad of udev so btrfs is neither in hooks or modules

# This file is part of https://github.com/random-archer/mkinitcpio-systemd-tool

# Ensure /sysroot mount before switch root. 
# See issue #10.
# See https://www.freedesktop.org/software/systemd/man/bootup.html

[Unit]
Description=Initrd SysRoot Mount
Documentation=https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/README.md
ConditionPathExists=/etc/initrd-release
ConditionPathExists=/dev/mapper/archroot-ROOT
ConditionPathExists=/sysroot
ConditionPathIsMountPoint=!/sysroot
DefaultDependencies=no
After=initrd-cleanup.service
After=initrd-root-fs.target initrd-fs.target initrd.target
Before=initrd-switch-root.service
Before=initrd-switch-root.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/mount -a 

[Install]
WantedBy=initrd-switch-root.service
WantedBy=initrd-switch-root.target

[X-SystemdTool]

# enable service
InitrdService=enable

The stangest part is.. i don't even have to start initrd-sysroot-mount. When i'm dropped to emergency shell i can simply start initrd-switch-root and everything works fine....

my guess is that initrd-switch-root is started before a mandantory service, but i have no idea wich one. The Bootlog is not very helpful either.

[Andrei-Pozolotin]

When i'm dropped to emergency shell i can simply start initrd-switch-root and everything works

you mean /sysroot is still/already mounted when you switch to emergency shell?

[kubax]

nope, it's not.. that's what is stange

[kubax]

Here is a pictue of my mounts after entering emergency shell.

[Andrei-Pozolotin]

1. likely this means initrd-sysroot-mount is started as dependency for initrd-switch-root

2. try to introduce a delay, to see if there is a race condition between
   unmount /sysroot initiated by systemd core vs mount /sysroot initiated by initrd-sysroot-mount
   https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/initrd-sysroot-mount.service#L23

ExecStartPre=/bin/sleep 5

3. attach here a zip of full boot log for review of service events ordering

journalctl -b | gzip  > boot.log.gz

[kubax]

try to introduce a delay, to see if there is a race condition between
unmount /sysroot initiated by systemd core vs mount /sysroot initiated by initrd-sysroot-mount
https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/initrd-sysroot-mount.service#L23

ExecStartPre=/bin/sleep 5

That did it....
So it seems to be a race condition like you allready guessed.

Whould it help to check if /sysroot is allready mounted and wait until it's unmounted? to let the boot process sleep for 5 second's is a litle "ugly" in my opinion :)

i'm going home now (it's on my work pc) so i could only test new patches tomorrow.

[Andrei-Pozolotin]

please test: try to insert one of:
https://github.com/random-archer/mkinitcpio-systemd-tool/blob/master/initrd-sysroot-mount.service#L15

a) service order

After=sysroot.mount

b) mutual exclusion

Conflicts=sysroot.mount

[kubax]

Nice.

After=sysroot.mount

did it!

System boots without the need of a delay.

Thank! 🥇 :)

[Andrei-Pozolotin]

in this case, please try simplified version

file:

initrd-sysroot-mount.service

content:

[Unit]
ConditionPathExists=/etc/initrd-release
DefaultDependencies=no
After=sysroot.mount
BindsTo=sysroot.mount

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/true

[Install]
WantedBy=initrd-root-fs.target

[X-SystemdTool]
InitrdService=enable

[kubax]

Also boots perfectly without errors :)

[Andrei-Pozolotin]

thank you

[Andrei-Pozolotin]

@kubax please verify v16 and close
https://aur.archlinux.org/packages/mkinitcpio-systemd-tool/

[kubax]

verified!

Works like expected. Thanks for your great work :)