LVM thin-provisioned volumes

[jpouellet]

It is my understanding that Qubes plans to moves to LVM thin provisioning by default in R4.0 (QubesOS/qubes-issues#2412), and this does not appear to be supported by Heads at the moment:

[osresearch]

Can you try turning it on in the config/linux.config file and seeing if it works?

diff --git a/config/linux.config b/config/linux.config
index 7ad8bab..14cc7f8 100644
--- a/config/linux.config
+++ b/config/linux.config
@@ -946,9 +946,11 @@ CONFIG_BLK_DEV_DM=y
 # CONFIG_DM_DEBUG is not set
 CONFIG_DM_BUFIO=y
 # CONFIG_DM_DEBUG_BLOCK_STACK_TRACING is not set
+CONFIG_DM_BIO_PRISON=y
+CONFIG_DM_PERSISTENT_DATA=y
 CONFIG_DM_CRYPT=y
 # CONFIG_DM_SNAPSHOT is not set
-# CONFIG_DM_THIN_PROVISIONING is not set
+CONFIG_DM_THIN_PROVISIONING=y
 # CONFIG_DM_CACHE is not set
 # CONFIG_DM_ERA is not set
 # CONFIG_DM_MIRROR is not set
@@ -2496,7 +2498,7 @@ CONFIG_CRC32_SLICEBY8=y
 # CONFIG_CRC32_SARWATE is not set
 # CONFIG_CRC32_BIT is not set
 CONFIG_CRC7=m
-CONFIG_LIBCRC32C=m
+CONFIG_LIBCRC32C=y
 CONFIG_CRC8=m
 # CONFIG_AUDIT_ARCH_COMPAT_GENERIC is not set
 # CONFIG_RANDOM32_SELFTEST is not set

[osresearch]

(unrelated to the LVM topic -- are you running Xen 4.6.1?)

[jpouellet]

Can you try turning it on in the config/linux.config file and seeing if it works?

After a while of rebuilding & reflashing Heads, and reinstalling Qubes with thin partitioning, same error. However, I didn't make clean, and none of the usual ways to read config of the running kernel seem to work, so it may be possible that I'm still running the old kernel? Rebuilding again to see if that might have been it... :/

For me to say that things actually work though, the documentation would need a bit more love. The intended qubes disk layout is unclear to me, and I've still not yet been able to successfully boot Qubes even once.

/bin/qubes-boot expects a /dev/qubes_dom0/00, but a default install only produces /dev/qubes_dom0/{root,swap}. However, if you select custom partitioning in the installer, the swap volume does get named 00. So you have some extra unnamed LV which you're calling root, but isn't root? I'm confused.

If I modify the script to get the blkid for the "root" LV instead of 00 (and use /bin/xen.gz instead of the one in /boot because otherwise I only ever get VGA rainbows) then I can kexec into the kernel & initramfs installed to /boot by qubes, which gives me a cryptsetup prompt and then sits at "Reached target Basic System." for several minutes before dropping me into a dracut recovery shell:

at which point I'm lost. (and rdsosreport.txt is an empty file)

I'm first trying to even see if I can even get Qubes to boot at all, nevermind the TPM and dm-verity... So far I've only been able to successfully boot the installer (via /bin/qubes-install).

(unrelated to the LVM topic -- are you running Xen 4.6.1?)

No, that's just what ships on the Qubes R3.2 install ISO. This is a dedicated machine I keep reinstalling.

[jpouellet]

Okay, so I'm an idiot. I had only rebuilt qemu.img, haha.

New failure mode:

Also, zooming in on https://www.flickr.com/photos/osr/33156102504/ is informative re: your partitioning scheme. Documentation in disguise :)

[jpouellet]

Okay, enough reinstalling for today. I'm going to go back to normal LVM and try to get this thing booting. Future tests should be possible with qemu.

[jpouellet]

In other (non-LVM-related) news, with:

• / on 00
• swap on 01
• /home on 02

Qubes boots! :)

[osresearch]

Finally getting around to installing a Qubes 4.0-rc1 after SHA and I think I see the problem...

With the old partitioning setup each device was its own luks entry, while with the thin provisioning there is only one partition and it is encrypted. To get to the volume group info the device needs to be mapped:

cryptsetup luksOpen /dev/sda2 qubes_dom0

Now lvm lvscan will show /dev/qubes_dom0/root (and swap and home).

however, we're missing /usr/bin/thin_check (from https://github.com/jthornber/thin-provisioning-tools) so other things are breaking.

[osresearch]

And the kexec-save-key script fails as well, since there is only one encrypted volume and it is trying to get the LUKS header for each partition in the volume group.

[flammit]

I've got another x230 coming arriving this week that I was planning to dedicate to Qubes 4/Heads testing. Will check out the LVM thin provisioning over the weekend.

[osresearch]

@jpouellet were you able to install Qubes 4 under Heads? I'm getting an error during the post install steps (split into issue #237)

[flammit]

Installed Qubes 4 rc1 using default settings (creates LVM thin provisioned pool in /dev/sda2) on the qubes-4 branch and everything seems to be working fine.

Looks like by default in a single LUKS crypt device (from lsblk in dom0):

sda (disk)
|--sda2 (part)
  |--luks-XXXX (crypt)
    |--qubes_dom0-pool00_tmeta (lvm)
...
    |--qubes_dom0-pool00_tdata (lvm)
...
    |--qubes_dom0-swap (lvm)
...

To boot as you would've in Qubes 3.2, during the Heads TPM disk key sealing you can just leave the LVM device blank and use just /dev/sda2 for the block device.

[tlaurion]

Even in Qubes 3.2 with latest patchwork, LVM must be blank and /dev/sda2 must be set as block device.

[flammit]

@tlaurion I think that's what I would expect, unless you had the LVM drive partitioning that @osresearch had originally outlined in the image in his docs (in which case you need the LVM == qubes_dom0 since each LVM volume was a LUKS crypt).

There's probably a better way to specify the LUKS device hierarchy / dependency, so I'd be open to suggestions for changes.

[tlaurion]

@jpouellet, @flammit, @osresearch @kylerankin : linked to heads-wiki issue I was going to update...

@jpouellet, @marmarek : any idea on what would be the good partition scheme and implementation to go forward? Heads could have lvm-thin-tools to facilitate measurements.... I'm just not sure of the way to go from here.

We have:

• TPMTOTP sealed secret from QR code. changing the clock is not really possible since the user validates the 6 digits on his phone. HOTP makes the validation with the key, makes it flash green and update /boot with a counter. Plus, the HOTP handshake result is shown on the screen as being valid. Changing the led to a green only LED is not a possible attack there.
• GPG public key being inserted in the rom, while the user is forced to sign with smartcard everytime his default boot config changed (grub.cfg, xen, initrd or kernel) is hard to circumvent also. Note that Heads generated a /boot sha256 digest on the fly at each boot, which is compared to what the user signed. If it changed, the user is warned of it. The absence of that file is of course also a red flag. Purism made a really good demo here.

What is missing is the possibility to measure templates between updates. I have no clue how to implement this right now until something is implemented in QubesOS. Would like to hear some inputs on where to go from now.

EDIT: original discussion happened here on qubes-devel.

[marmarek]

@jpouellet, @marmarek : any idea on what would be the good partition scheme and implementation to go forward? Heads could have lvm-thin-tools to facilitate measurements.... I'm just not sure of the way to go from here.

I don't think thin-provisioning is needed within heads. Boot files should be in separate partition (not even LVM). What case I'm missing?

[tlaurion]

@jpouellet, @marmarek : any idea on what would be the good partition scheme and implementation to go forward? Heads could have lvm-thin-tools to facilitate measurements.... I'm just not sure of the way to go from here.

I don't think thin-provisioning is needed within heads. Boot files should be in separate partition (not even LVM). What case I'm missing?

@marmarek : To measure templates that are desired to stay RO, just like intended here in previous discussion by the help of dm-verity. And to start the discussion on how to make dom0 measurable.

[marmarek]

If going with measurable dom0, I'd simply put it outside of LVM thin pool, as a static LVM volume. That would have other benefits as well, like safety against filling up thin pool. The downside is wasting some of disk space - you need an extra space in dom0 that is rarely used (unpacking templates, temporary storage for backups etc) - with static LVM volume, you can't reuse it for running VMs. But if going for R/O dom0, that would need to be elsewhere anyway, so it shouldn't be an issue.

Then, having measured dom0, you can leave measuring templates to that dom0.

[tlaurion]

I think adding thin-provisioning-tools in Heads is be the first step torward temmplates integrity validation to permit QubesOS being preinstalled without having faith (vs verifiable trust) in OEM/Organization QubesOS predeployments.

My main interest here is to provide a way for the user to validate that the OEM cloned disk image is in the same state that the OEM intended (if it could be in the same state that QubesOS intended, that would be even better!).

• Right now, the only tool available is something like rpm -qa|while read line; do echo $line:; sudo rpm -V $line; done in dom0 and validate differences manually. (and templates equivalents)

• The OEM could sign thin volumes integrity prior to shipment and user reownership, with key inserted and measured in PCR7.

With that, the user would be able to validate that:

• Public key in rom is the one sent over secure communication channel
• ROM integrity matches TOTP code generated on smartphone with shared and scanned Qr code, confirmed automatically with Librem Key flasshing green and "HOTP: OK" statement prior to continuing with reownership
• /boot integrity was signed with OEM corresponding private key (matching public key measured in rom with PCR7)
• LVM filesystems are in the state intended by the OEM(good enough)/QubesOS(best option).
  • @marmarek : I think sparsebak project is making a big step torward this goal/in the good direction. The OEM/QubesOS could provide a repository where the user could re-deploy/update its template to a reference, being in an up-to-date state, reverting user customization and software updates if desired or updating only relevant part, minimizing bandwidth and maximizing trust on the state of its templates.

Thoughts?

[tlaurion]

@flammit @osresearch and others: here is my WiP.
Can't get thin-provisioning to compile against musl-cross, while dependencies seems to build correctly. Insights?

[tlaurion]

Thinlvm can be opened without added tools for a while.