generate-zbm EFI bundle doesn't kexec into kernel, prebuilt EFI kexecs fine

[Contik]

Hi all,

I have an Arch Linux system with ZFSBootMenu running. It works perfectly fine with the prebuilt EFI release image downloadable at https://get.zfsbootmenu.org/efi. However, I can't get a generate-zbm EFI bundle to kexec into my kernel and initramfs; only the prebuilt image can do that.

When I boot into my self-compiled EFI bundle ZFSBootMenu doesn't seem to correctly hand control over to the ZFS dataset's kernel and initramfs and instead reboots the machine. With the prebuilt EFI release image no such reboot happens, that one just works perfectly.

I guess this post boils down to: what makes the prebuilt EFI release image kexec into my kernel and initramfs that I can't replicate with the self-built EFI bundle? I'm assuming kernel and initramfs files are perfectly fine since they do work when paired with the prebuilt EFI release image. Even my self-compiled image can see my kernel and initramfs, it just doesn't boot into them. Is there some debugging I can do to better understand what exactly is causing a reboot where the prebuilt EFI release image instead does a clean kexec?

I'd appreciate a second pair of eyes pointing me at what I'm missing. Thanks!

Here's what I have:

• Arch Linux with latest zfsbootmenu package (releases/tag/v2.3.0 from this repo)
• initramfs gets created via mkinitcpio with dependencies met per docs:
  
• Disk is GPT-partitioned
• Dual boot Windows/Linux setup
• File /etc/zfsbootmenu/config.yaml has:

  Global:
    ManageImages: true
    BootMountPoint: /efi
    InitCPIO: true
  Components:
    Enabled: false
  EFI:
    ImageDir: /efi/EFI/ZBM
    Versions: false
    Enabled: true
  Kernel:
    CommandLine: ro loglevel=0 zbm.import_policy=hostid
    Prefix: vmlinuz
  

• File /etc/zfsbootmenu/mkinitcpio.conf has:

  # sed -e '/^#/d' -e '/^$/d' /etc/zfsbootmenu/mkinitcpio.conf
  MODULES=()
  BINARIES=()
  FILES=()
  HOOKS=(base udev autodetect modconf block filesystems keyboard)
  

• EFI System Partition is mounted at /efi where /efi/EFI/ZBM/vmlinuz.EFI sits next to Windows Boot Manager files:

  /efi
  ├── EFI
  │   ├── Boot
  │   │   └── ...
  │   ├── Microsoft
  │   │   └── ...
  │   └── ZBM
  │       └── vmlinuz.EFI
  ├── ...
  └── ...
  

• The one and only existing mkinitcpio preset file at /etc/mkinitcpio.d/linux.preset has:

  ALL_kver="/boot/vmlinuz-linux"
  ALL_microcode=(/boot/*-ucode.img)
  PRESETS=('default')
  default_image="/boot/initramfs-linux.img"
  

• File /etc/mkinitcpio.conf has:

  # sed -e '/^#/d' -e '/^$/d' /etc/mkinitcpio.conf
  MODULES=()
  BINARIES=()
  FILES=(/etc/zfs/zpool.key)
  HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont numlock block zfs filesystems)
  

• ZFS zpool has bootfs property set:

  # zpool get bootfs
  NAME   PROPERTY  VALUE                     SOURCE
  zpool  bootfs    zpool/root/archlinux-frn  local
  

• Where dataset zpool/root/archlinux-frn is currently the only root boot environment
• Boot environment is encrypted and has org.zfsbootmenu:commandline set like so:

  # zfs get encryptionroot,keylocation,encryption,org.zfsbootmenu:commandline zpool/root/archlinux-frn
  NAME                      PROPERTY                     VALUE                                       SOURCE
  zpool/root/archlinux-frn  encryptionroot               zpool                                       -
  zpool/root/archlinux-frn  keylocation                  none                                        default
  zpool/root/archlinux-frn  encryption                   aes-256-gcm                                 -
  zpool/root/archlinux-frn  org.zfsbootmenu:commandline  rw nowatchdog rd.vconsole.keymap=de-latin1  local
  

• The keylocation in my encryptionroot is set as:

  # zfs get keylocation zpool
  NAME   PROPERTY     VALUE                      SOURCE
  zpool  keylocation  file:///etc/zfs/zpool.key  local
  

• Where mkinitcpio -P will embed the same zpool.key file into initramfs via /etc/mkinitcpio.conf
• One plain regular linux kernel and an initramfs live at /boot inside the encrypted zpool

  /boot
  ├── amd-ucode.img
  ├── initramfs-linux.img
  └── vmlinuz-linux
  

• lsinitcpio confirms that the key file lands in my initramfs:

  # lsinitcpio /boot/initramfs-linux.img | grep 'etc/zfs/zpool.key'
  etc/zfs/zpool.key
  

[zdykstra]

If the pre-built EFI can boot your system / kernel /initramfs, there's very little that points to this being an issue with your BE's initramfs configuration, so basically all of that configuration (for now) can be ignored.

I'd recommand adding loglevel=7 to the org.zfsbootmenu:commandline property to see if that shows anything on your screen. Barring any output from that, can you share your locally built vmlinuz.EFI ? There shouldn't be anything sensitive in there.

One last note, zbm.import_policy=hostid is the default import policy, there's no real need to explicitly specify it on the EFI commandline.

[Contik]

Adding loglevel=7 to the boot environment's dataset org.zfsbootmenu:commandline property unfortunately doesn't shine a brighter light on the situation. I see two lines of USB teardown messages in both cases when boot continues after I've entered and confirmed by dataset password. With a prebuilt image kexec seems to just work immediately afterwards (and console log output becomes appropriately verbose) while with my own image I just see a reboot after these lines instead:

I've uploaded my self-compiled EFI image to http://0x0.st/HRhD.tar. It'll extract as a single file named vmlinuz.EFI.borked. The upload will expire end of March.

Just for cleanliness I've removed the zbm.import_policy=hostid from generate-zbm's config.yaml setting .Kernel.CommandLine. Thanks for that tip! I've also experimented with setting ZFSBootMenu's own loglevel=7 just for fun but couldn't detect any log content whatsoever inbetween USB teardown messages showing up and the system rebooting.

A bit on the esoteric side of things: since these notices about USB controller teardown are the last thing I see with my own vmlinuz.EFI before a reboot I unplugged a bunch of USB peripherals and even replaced keyboard and mouse with a different one to no avail. Just saying :)

[zdykstra]

My initial testing here hasn't revealed anything obviously wrong. The EFI has stock ZFSBootMenu 2.3.0 built on top of recent Arch tooling. Booting it in a VM, I'm able to successfully boot both an Arch and a Void environment.

One possible source of an issue is that the binary release is using Linux 6.1.x, whereas your local build is using 6.7.x. If possible, maybe you can try building a local image on top of the linux-lts kernel (what does Arch point that at now?) to see if that changes anything.

[Contik]

Funnily enough this has helped my understanding. I switched from kernel linux to linux-lts which at this time points to version 6.6.x. I went with the next-oldest branch 6.1.x to closely emulate what the prebuilt EFI image is doing.

• 👍 With linux-lts 6.1.71 running I compiled an EFI image (see 0x0.st/H7X2.x.tar, will extract as vmlinuz.EFI.lts-6.1.x) which was able to kexec into the same linux-lts 6.1.71 kernel.
• 👍 As next step I upgraded just the installed kernel from linux-lts 6.1.71 to newest LTS branch 6.6.19 and left the ZFSBootMenu EFI file untouched. The 6.1.71 EFI image was still able to kexec me into linux-lts 6.6.19.
• 👎 Lastly with linux-lts 6.6.19 running I rebuilt the ZBM EFI image (0x0.st/H7XV.x.tar, extracts as vmlinuz.EFI.lts-6.6.x) and it was not able to kexec me into my linux-lts 6.6.19 kernel.

I switched my EFI image back to the one I built before with linux-lts 6.1.71 and switched my OS back to linux 6.7.6. I'll happily do more tests in case you have an idea what I could do here. For now I at least know that I have a way out if I shoot myself in the foot :)

[ahesford]

There was some shakeup with kexec and the EFI stub around kernel 6.2. I wonder if this is related, and if @classabbyamp can shed some light on that issue.

[classabbyamp]

i don't think kexec was related to the 6.2 stub change (CONFGI_EFI_HANDOVER or something like that), and on arch the stub used is probably systemd-boot's, which would not be affected by the change

[zdykstra]

Another user on IRC has reported an issue incredibly similar to this. Using a locally built EFI on Void Linux with linux-6.7.9, linux-6.6.21 and linux-6.1.79, their system immediately reboots after the BE's kernel is kexec'd. The 2.3.0 binary release with linux-6.1.63 works for them.

[TinfoilSubmarine]

I have also run into this issue. When using a locally built EFI On Void Linux with linux-6.6.21 I get the same behavior mentioned above where directly after the "Tearing down USB controller" messages, my system reboots. Curiously, I can only reproduce the issue on one of my Void Linux machines--both machines have linux-6.6.21.

[zdykstra]

That's really disconcerting to hear. That, at least to me, points to this possibly being some hardware specific to a machine. Are you able to provide a few details on the problematic machine (make/model/output of dmesg in a pastebin) ?

[TinfoilSubmarine]

It's a custom build. ASRock B450M Pro4-F mothboard, Ryzen 2600, and Radeon RX 590.

https://gist.github.com/TinfoilSubmarine/c25510e917948aff3b85e77764d4015a

[Contik]

Mine's also an AMD socket AM4 mainboard, an MSI B550-A Pro (model 7C56-002R). See also output of Arch Linux inxi -F at http://0x0.st/HFWy.txt. I have since switched from linux to linux-zen kernel for reasons unrelated to this discussion, other than that same system as when I posted. I'll see if I can get a sufficiently different machine set up the same way for a test with root-on-ZFS and ZFSBootMenu.

[Contik]

I've set up a comparison laptop where this works fine: the machine runs Arch Linux with linux kernel version 6.7.9, it can generate-zbm and then after a reboot kexec into that same kernel and initramfs combination without issue. Specs per inxi -F at http://0x0.st/HCJe.txt, output of dmesg -T at http://0x0.st/HCOG.txt.

The laptop has an AMD socket FP6 with a preassembled Ryzen 7 7730U, it's an HP ProBook 455 G10.

Config components:

• Same as above
  • File /etc/zfsbootmenu/config.yaml
  • ZFS zpool bootfs property, other than that the boot environment has a slightly different name
  • Only one boot environment exists
  • ZFS property org.zfsbootmenu:commandline
  • zpool keylocation property
  • Contents of /boot
• Different from above
  • File /etc/zfsbootmenu/mkinitcpio.conf
    • Also has HOOKS=(... net dropbear), I was playing around with Dropbear SSH in ZFSBootMenu but stopped those experiments. As of right now /etc/zfsbootmenu/config.yaml has no more net in its .Kernel.CommandLine so it's identical to the desktop above.
    • //edit a few minutes after posting: Just to rule this out I've removed HOOKS=(... net dropbear) from /etc/zfsbootmenu/mkinitcpio.conf and rebuilt the EFI image. generate-zbm --debug confirms that net and dropbear are no longer built into the image. EFI image still kexecs perfectly fine.
  • EFI System Partition residents
    • Some HP laptops nowadays apparently ship with what HP calls their HP PC Hardware Diagnostics UEFI, that's 130 MiB in /efi/EFI/HP, rest is dual boot Windows/Linux as above.

          /efi
          ├── EFI
          │   ├── Boot
          │   │   └── ...
      --> │   ├── HP
          │   │   └── ...
          │   ├── Microsoft
          │   │   └── ...
          │   └── ZBM
          │       └── ...
          ├── ...
          └── ...
      

  • File /etc/mkinitcpio.d/linux.preset
    • Line ALL_microcode=(/boot/*-ucode.img) is gone since mkinitcpio v38 deprecated its use.
  • File /etc/mkinitcpio.conf
    • Now has

      `HOOKS=(base udev autodetect microcode ...)
                                   ~~~~~~~~~
      

      This is the new way of early loading microcode files and goes hand in hand with the change above. Just for fun I went with the old way as well. It still works and just triggers a warning message. EFI image can still kexec perfectly fine either way on this laptop. And also just for fun I made the same change on my desktop, behavior there did not change, i.e. an EFI image built with a recent kernel does not kexec into a recent kernel no matter how the microcode hook comes into play.

[ahesford]

I can reproduce this on my desktop using newer linux6.1 as well as linux6.6. In the 6.1 series, the problem was introduced by version 6.1.79. I'm not sure which versions of 6.6, if any, may have worked.

Until we can uncover more information, I advise Void users interested in custom ZBM images to install linux5.15 and linux5.15-headers, then edit your /etc/zfsbootmenu/config.yaml to specify

Kernel:
  Version: 5.15.*

At least the current version of linux5.15, 5.15.151, seems to work fine on my hardware.

I'm converting this to a proper issue: #598.

[slymattz]

Thanks. I'm running Void Linux and what you proposed gets my AM4 platform to boot 👍

[ahesford]

A fix is in sight: I bisected Linux 6.1 to identify the commit that breaks booting with ZBM, and can confirm that this is the same problem described by others and fixed upstream with a revert. Assuming that my testing confirms that backporting this fix to Void fixes the issue, Void users will once again be able to create custom images with their preferred kernel series.