Debian-12 still requires manual crypttab alterations and initrd regeneration to pick up TPM disk unlock key

[tlaurion]

originally posted under #1473 (comment) by @natterangell

Debian still asks for the password when booting though (in addition). Is that expected behaviour? I do have encrypted swap, not sure if that affects it(?)

It all depends on the partition scheme. If using BRTFS (same as if using BRTFS under Qubes 4.1+ partition scheme) then two LUKS containers are expected to be unlocked from the TPM disk unlock key. So you should go back into Boot options -> show boot options -> save as default, and answer yes to reseal TPM disk unlock key. But this time, pass the two LUKS containers (eg /dev/sda2 /dev/sda3). Qubes default TLVM partition scheme has it all under a single LUKS container. I guess the information under https://osresearch.net/InstallingOS/#compatibility is not specific enough and could be improved, or the code could suggest found LUKS partitions only to confuse less the user.... Note that BRTFS is not supported within Heads. Only ext4 and TLVM (Thin LVM volume support without thin-lvm-tools). If needs be and tangent is to go torward XFS and BRTFS, we could try to add those tools as well. Note that #1446 is adding support for EXT4 repair tools and ExFat repair tools as well which is now the default for bought sdcards and thumb drives, where Thin LVMs are more and more despised for their added complexity which slows down disk operations.

Also, I'm not sure if https://osresearch.net/InstallingOS/#default-boot-and-disk-unlock is true anymore, since my testing with Talos II and debian invalidates the statement there saying that init doesn't process crypttab: it does, otherwise I would not be able to boot into my OS. @krystian-hebel you reported something similar. What is your partition scheme on Talos II's debian install?

The magic happening in the background here is:

• Heads tries to extract OS initrd and find crypttab there.
• if found, it reuses the UUID there and overrides the config pointing to LUKS key for that entry(ies)
• It creates a secret.cpio containing crypttab override file and the secret used to unlock the LUKS container(s)
• It kexec to the final OS, passing that additional cpio in memory to be extracted on top of initramfs memory
• The OS goes through its normal init script, checks crypttab and attempts to use exposed crypttab file.

@natterangell : So if only one LUKS container is overriden out of many, the final OS will ask for as many LUKS container it doesn't know how to unlock. Let me know if specifying both LUKS containers when setuping a TPM disk unlock key worked (it should: the logic parses those entries and will fail if the partition provided is not a LUKS drive, otherwise leaving slot 0 untouched (disk recovery key) and kills slot1 and then replaces slot 1 with the in-memory generated key corresponding to sealed secret in TPM, which TPM disk unlock key is unsealed with sealed measurements and NV region passphrase. This is why you type a different passphrase on Heads (TPM NV region passphrase to unlock TPM disk unlock key) then on Final OS (Disk recovery key). Hope this clarified everything. Would appreciate if you could do a PR on heads-wiki if that information is unclear to you.

---

Related: asked question to @marmarek at https://forum.qubes-os.org/t/btrfs-and-qubes-os/6967/28 to see if BRTFS was expected to become the default partition scheme over current TLVM default, since QubesOS is first citizen of Heads. But if all OSes are moving away of TLVM, Heads will need to choose what to support in kernel and on filesystem tools provided for recovery shell. Heads cannot support everything since we are limited in ROM space (until we can switch to squashfs/overlayfs, when: undefined).

[tlaurion]

@natterangell please update this issue. Discussions under PR should be limited to changes in code, not discussing issues.
#1472 is now closed because #1473 has been merged and this is to discuss ecosystem moving to other filesystems, OS installer defaults and what Heads should have in its recovery shell support.

@marmarek : Will Q4.2 move to BRTFS? (cross-post https://forum.qubes-os.org/t/btrfs-and-qubes-os/6967/28)

[natterangell]

Well exFAT support would certainly be appreciated. I suppose most people using Heads mostly use linux, but windows won't allow FAT32 partitions on larger usb drives (at least not without some fiddling), and so adding rom files, gpg keys etc requires preparing the drive on linux/macos

[tlaurion]

Cross-linking to QubesOS/qubes-issues#6476

[tlaurion]

@natterangell can you detail Debian version and existence non-existent of crypttab and final content (hide like UUID) here? I'm glad PR worked for you but issue should not be there, bit here to track it.

[tlaurion]

@natterangell can you detail Debian version and existence non-existent of crypttab and final content (hide like UUID) here? I'm glad PR worked for you but issue should not be there, bit here to track it.

He answered

Debian 12 "Bookworm", default encrypted install with LVM.

As mentioned here:

(*) Ubuntu/Debian Note: These systems don’t read /etc/crypttab in their initrd, so you need to adjust the crypttab in the OS and update-initramfs -u to have it attempt to use the injected key. Due to oddities in the cryptroot hooks, you also need keyscript to be in /etc/crypttab even as a no-op /bin/cat:

sda5_crypt UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX /secret.key luks,keyscript=/bin/cat_

The vanilla file as installed by Debian looks like this:

sda5_crypt UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX none luks,discard

Replacing "none" with "/secret.key", and adding "keyscript=/bin/cat" at the end of the line as described in wiki, then update-initramfs -u and resigning/reseal in Heads unlocks the disk without the need to re-enter password during boot.

@natterangell weird. Location of the file created is under /etc/crypttab or somewhere else? As said in the pr/original issue (discussion should continue here to track issue down) qubes doesn't provide crypttab either, but when provided by heads under secret.cpio, it complies reading the one provided.

This means that the initnif Debian looks for it, but maybe somewhere else? Heads definitely provides one. Will try to replicate but it would be nice to understand what is missing in the one crafted by Heads since Debian is pretty common. nevermind. Noop. Weird. This is a case where Heads usage requires wizardry which is not yet user friendly enough....

[natterangell]

I apologize for commenting on the PR.

Yes, the default on Debian is /etc/crypttab.

The Debian manpage for crypttab states:

If the key file is the string none, a passphrase will be read interactively from the console.

I don't know if update-initramfs checks this field and includes the file in initrd if something else than "none"?

[natterangell]

The manpage states further down that:

This option is specific to the Debian crypttab format. It's not supported by systemd.

WARNING: With systemd as init system, this option might be ignored. At the time this is written (December 2016), the systemd cryptsetup helper doesn't support the keyscript option to /etc/crypttab. For the time being, the only option to use keyscripts along with systemd is to force processing of the corresponding crypto devices in the initramfs. See the 'initramfs' option for further information.

Seems there might be some differences despite both Debian and Fedora/Qubes being systemd-based

[tlaurion]

Sorry if I was unclear.

If a crypttab is found under initrd, heads parses it ad overwrites none by /secret.key and adds discard option.

You should see those happening when setting a default option and enabling tpm disk unlock key. It says if it found one, says what he found and what is replaced by, and if not, what has been generated. There is even an override file under /boot to say what is the override for transparency, which is signed and used at time of generating the secret.cpio passed by the kexec call.

The only thing I see here is the default generated initrd not containing /etc/crypttab and Heads doing the wrong thing as opposed to what their crypt-mount is expecting... Seems like we have a forever unfixed thing under Debian.

I guess I'll have to reproduce and report upstream..... Damn.

@natterangell can you point me to the direct iso you have used to install Debian please?

Also, we are talking about master romnfor your board I guess right? This code has changed a couple of months ago and I remember Debian was specifically tested. Now I'm really confused.

Logic to parse initrd and override crypttab when found: https://github.com/osresearch/heads/blob/d8a9a1e77e1731302e5b464b789c872a78e738c4/initrd/bin/kexec-save-default#L141

Logic to generate crypttab when none is found under initrd: https://github.com/osresearch/heads/blob/d8a9a1e77e1731302e5b464b789c872a78e738c4/initrd/bin/kexec-insert-key#L94

[natterangell]

I used debian-12.1.0-amd64-netinst.iso. As of Debian 12, firmware is now included in the default image.

Yes, before manually editing the default crypttab file, the following is displayed:

That message goes away after manual edit.

[tlaurion]

I used debian-12.1.0-amd64-netinst.iso. As of Debian 12, firmware is now included in the default image.

Yes, before manually editing the default crypttab file, the following is displayed:

That message goes away after manual edit.

This means that Debian doesn't ship with crypptab on initrd at install, and then when you provide one on the host and regenerate initrd, it's now packed under /boot.

I have reinstalled my qemu disk and will be able to reproduce eventually. This could be fixed upstream: it makes no sense that the crypttab is inside of the rootfs: it should be part of initrd under /boot to be parsed by anyone who wants to ans overriden on the OS rootfs if needed.

It seems that a lot of OSes are doing it wrong.
NixOS and now Debian. Quebesos doesn't provide one, but deals with the one generated happily without needed illogical stuff in it.

Will try to replicate but that would need upstreaming. It's not Head job to fix downstream, even less if Debian is used by other repackaged OS distributions.

Ideal would be for you to check the init code and follow the rabbit there and open an issue and refer it here. Documenting temporary fixes I'm ok, maintaining them I'm not. Forgetting about it and searching again in the future, a total frustration. This is open source and if spotted on, they should want to fix it.

Up to task @natterangell ?

[marmarek]

@marmarek : Will Q4.2 move to BRTFS? (cross-post https://forum.qubes-os.org/t/btrfs-and-qubes-os/6967/28)

No. We may consider it for some later release, but not 4.2

[natterangell]

Ideal would be for you to check the init code and follow the rabbit there and open an issue and refer it here. Documenting temporary fixes I'm ok, maintaining them I'm not. Forgetting about it and searching again in the future, a total frustration. This is open source and if spotted on, they should want to fix it.

I'll give it a shot.

[tlaurion]

@natterangell Sorry for the delay.

Trace of tests while testing #1482 on qemu-coreboot-whiptail-tpm2 with #1479 equivalent being enabled to enable debug output and tracing (enabled by default in board configs on qemu boards aimed at testing) on TPM Disk Unlock Key enabled default boot:

And here I confirm that previous paramenters are not taken into consideration from Debian's initrd, since Disk Recovery Key passphrase is prompted:

---

Will edit this post with investigation. But My hunch is that the system only needs to be updated and that once host's /etc/crypttab will be injected into initrd, Heads will do the right thing. Let's see...

EDIT: stalled there for like 10 minutes in QEMU in TCG mode (as opposed to KVM, since under QubesOS and nested virt is not avail and broken @marmarek: this makes QubesOS as a testing platform less then adequate, would be nice if given as option. Can you link ticket to track on Xen side or something? )

---

On reboot. Of course, /boot changed:


















So no. The override as done currently by Heads is not enough.

Let's go back to what was packed inside of initrd without any manual modification of the host's crypttab file and go through above screenshots to see what is happening.

[tlaurion]

So to get back to basics:

/boot # cat initrd.img-6.1.0-11-amd64  | cpio -t
.
kernel
kernel/x86
kernel/x86/microcode
kernel/x86/microcode/AuthenticAMD.bin
107 blocks
/boot # cat initrd.img-6.1.0-10-amd64  | cpio -t
.
kernel
kernel/x86
kernel/x86/microcode
kernel/x86/microcode/AuthenticAMD.bin
107 blocks

No crypttab injected into initrd by debian whatsoever? I would have expected the crypttab to have been injected inside of the initrd here, i'm confused to how your fix has worked. What is the content of your /boot/initrd here @natterangell ?

[tlaurion]

From Heads:

cryptsetup open /dev/vda5 test 
mount /dev/mapper/debian--vg-root /media
cat /media/etc/crypttab 
vda5_crypt UUID=735e5c24-efe4-4869-b2f6-867f185a51ca none luks,discard

Provided by heads:

+++ The following /etc/crypttab lines will be passed through cpio into kexec call for default boot option: 
luks-735e5c24-efe4-4869-b2f6-867f185a51ca UUID=735e5c24-efe4-4869-b2f6-867f185a51ca /secret.key luks, discard

[marmarek]

/boot # cat initrd.img-6.1.0-11-amd64 | cpio -t

Don't use just cpio -t, you only get earlyinitrd this way, use lsinitrd tool.

[natterangell]

@tlaurion I’ll check later tonight when I have access to the machine.

while looking into this I came across a post on Stack Exchange indicating that

After more research, I can now finally answer the original question: /etc/crypttab is not present in initramfs because the default unlock script does not use that location; it uses /cryptroot/crypttab instead

and indeed, that seems to be the case. Still doesn’t explain why Heads succeeds after the manual edit described in the wiki above.

[tlaurion]

/boot # cat initrd.img-6.1.0-11-amd64 | cpio -t

Don't use just cpio -t, you only get earlyinitrd this way, use lsinitrd tool.

Thanks for the tip @marmarek , i'm afraid those are not provided by busybox once again....

~ # ls
ls      lsattr  lsmod   lsof    lspci   lsscsi  lsusb

What I hear is that there is more then what I see. I will try to find alternatives, thanks

[tlaurion]

@tlaurion I’ll check later tonight when I have access to the machine.

while looking into this I came across a post on Stack Exchange indicating that

After more research, I can now finally answer the original question: /etc/crypttab is not present in initramfs because the default unlock script does not use that location; it uses /cryptroot/crypttab instead

and indeed, that seems to be the case. Still doesn’t explain why Heads succeeds after the manual edit described in the wiki above.

@natterangell I like to read that. That is actionable and redirects what where to work to have a generic solution here.

@marmarek QubesOs happily complies with how Heads feeds crypptab content into secret.cpio after parsing only earlyinitrd.

Even if we were to feed the same crypttab content to /cryptroot/crypttab (easy) it seems that the format of the file should be more generic and provide the name of the vg here. I will hack my current working branch to feed cryptroot/crypttab here and see what happens. Otherwise this will require not reading outside sources, but looking at what the actual code does and what format it happily parses.

[marmarek]

Qubes should also support rd.luks.key=... argument (no crypttab change is necessary then). Maybe Debian has something similar too?

[natterangell]

Thanks to @marmarek for mentioning that tool! @tlaurion, it is indeed in cryptroot/crypttab

root@x230:~# lsinitrd /boot/initrd.img-6.1.0-11-amd64 | grep crypttab
-rw-r--r--   1 root     root           97 Aug 30 20:32 cryptroot/crypttab

[natterangell]

lsinitrd is in dracut-core on debian, btw.

[natterangell]

For reference, with cpio:

root@x230:~# cat /boot/initrd.img-6.1.0-11-amd64 | cpio -t
kernel
kernel/x86
kernel/x86/microcode
kernel/x86/microcode/.enuineIntel.align.0123456789abc
kernel/x86/microcode/GenuineIntel.bin
14064 blocks

[natterangell]

Reading the almost 8 year old(!) blog post that is mentioned in the wiki, I think I now understand why those changes work:

So far, so good. The crypttab man page even talks about how you can point to a keyfile in the third column (“none” above). Unfortunately, if you do so, it doesn’t actually work. If you read the cryptroot script2, you find that the keyfile is only ever used as an argument to a keyscript.

Oddly enough, although there are several provided scripts, each doing various exotic things, none of them seem to handle the simple case of using the key to just decrypt the drive.

What these scripts have in common is that they take the “keyfile” as an argument, and their output is used as the key. So, all we need to do is provide a “script” that will take a filename and output the file’s contents:

lvm /dev/sda1 /crypto_keyfile.bin luks,keyscript=/bin/cat
Now, the cryptroot hook3 will copy the cat executable into the ramdisk, and during boot cat will send the keyfile’s contents to cryptsetup.

So the workaround mentioned in the wiki basically makes Debian cryptroot hook script cat the "secret.key" injected by Heads to cryptsetup.

[natterangell]

@tlaurion

Even if we were to feed the same crypttabl to /cryptroot/crypttab (easy) it seems that the format of the file should be more generic and provide the name of the vg here

Wouldn't it be tricky to find the vg name, given that debian generates that based on whatever is chosen as hostname during install. But the UUID should be able to be extracted, no (?)

[tlaurion]

Reading the almost 8 year old(!) blog post that is mentioned in the wiki, I think I now understand why those changes work:

So far, so good. The crypttab man page even talks about how you can point to a keyfile in the third column (“none” above). Unfortunately, if you do so, it doesn’t actually work. If you read the cryptroot script2, you find that the keyfile is only ever used as an argument to a keyscript.

Oddly enough, although there are several provided scripts, each doing various exotic things, none of them seem to handle the simple case of using the key to just decrypt the drive.

What these scripts have in common is that they take the “keyfile” as an argument, and their output is used as the key. So, all we need to do is provide a “script” that will take a filename and output the file’s contents:

lvm /dev/sda1 /crypto_keyfile.bin luks,keyscript=/bin/cat
Now, the cryptroot hook3 will copy the cat executable into the ramdisk, and during boot cat will send the keyfile’s contents to cryptsetup.

So the workaround mentioned in the wiki basically makes Debian cryptroot hook script cat the "secret.key" injected by Heads to cryptsetup.

Houla... A bunch of hacks to hack other hacks...

@tlaurion

Even if we were to feed the same crypttabl to /cryptroot/crypttab (easy) it seems that the format of the file should be more generic and provide the name of the vg here

Wouldn't it be tricky to find the vg name, given that debian generates that based on whatever is chosen as hostname during install. But the UUID should be able to be extracted, no (?)

That part is clean and should work. Trying to simply propagate the same content in both cryptroot/crypptab and /etc/crypttab should be sufficient. Tried to get cpio from busybox do the right thing under Head's busybox without success so ideal here would be to simply have debian pickup to whatever it deems fine, where luks-uuid is the generic format and normally considered to be valid... I'm on it.

[tlaurion]

Let's see....

And it works!

Cleaning code...

[tlaurion]

@natterangell I would prefer the code to override what is inside of the initrd instead of creatinc generic options.
Goal of this ~new codepath was to actually override the OS desired options for LUKS, not to craft them but in last resort.

@JonathonHall-Purism it seems we overlooked that in previous attempts of fixing TPM Disk Unlock Key (which is not used by neither Purism/Nitrokey today) and since it was working out of the box for QubesOS Heads users, that fell in the cracks.

Small investigation on my side shows cpio is not behaving correctly neither in extracting nor listing files, stopping at early cpio and not processing further. I will try to fix this further, but as of now, I think this is way better than it was, with a lot of usability fixes that will land under #1482, including fix to close this issue!

[JonathonHall-Purism]

Small investigation on my side shows cpio is not behaving correctly neither in extracting nor listing files, stopping at early cpio and not processing further. I will try to fix this further, but as of now, I think this is way better than it was, with a lot of usability fixes that will land under #1482, including fix to close this issue!

@tlaurion I mentioned this over on #1482 as well, but I think what you want is initrd/bin/unpack_initramfs.sh.

Debian 12 provides an early microcode update in the initrd, which must be in an initial uncompressed segment. The real initrd follows in a zstd-compressed segment. Linux allows any arbitrary sequence of compressed/uncompressed segments concatenated together in the initrd.

unpack_initramfs.sh unpacks such an initrd, with a few limitations due to gunzip/unzstd (once it hits a compressed segment, no further segments can be read, because gunzip/unzstd can't be convinced to stop reading at the end of their compressed data). In practice, this has worked for the initrds I have tested, I use it for the Librem Blob Jail. It should be able to pull out the crypttab.

[tlaurion]

@natterangell I don't get it #1482 (comment)

More specifically that thread referred images.

When default boot TPM DUK is setuped:

When default boot is launched:

Debian fails even earlier. Spent too much time on that... Only insight is from last image: it seems that the path for crypttab is not cryptroot/crypttab but tmp something somehting. Will check one last time...

[tlaurion]

@natterangell yep. That's it:

~ # cat /boot/kexec_initrd_crypttab_overrides.txt 
/tmp/initrd_extract/cryptroot/crypttab:vda5_crypt UUID=735e5c24-efe4-4869-b2f6-867f185a51ca /secret.key luks,discard

Checking...