QEMU testing with OS, TPM, USB token

[JonathonHall-Purism]

Update: summary of final state:

Heads works in Qemu with a software TPM and USB-forwarded token; added a new config qemu-coreboot-fbwhiptail-tpm1-hotp.

With this config it is possible to:

• Provision TPM and TOTP/HOTP
• Sign /boot
• Actually boot the OS
• Persist all of the above, so we can test scenarios like:
  • Update (or tamper with) /boot and verify behavior
  • Update (or tamper with) firmware and verify behavior
  • Resetting TPM/TOTP/HOTP, etc.

Live OS installers work, Debian netinst works (tested installing XFCE, others should work). Lightweight desktops are essential under Qubes due to lack of KVM acceleration. Disabling memory ballooning in the qube is also important.

Heads requires unencrypted /boot, so Debian must be partitioned manually (Debian 11 defaults to encrypted /boot).

The remaining key limitation is that Heads cannot flash itself from qemu, so a GPG key cannot be added, for now this is done at build time using instructions in README.md. Re-ownership and OEM reset can't be fully tested yet due to this limitation. This PR is still useful for testing a lot of Heads, so this can be a future improvement.

There are two key problems before flashing would work:

• The Heads image must be mapped as pflash (not "bios", these are alternatives in qemu), which means it must fit in 8 MB. (Qemu limits flash to 8 MB currently, might be easy to increase but there are also supported boards with only 8 MB, so it's probably best to optimize the kernel config to fit in 8 MB.)
• flashrom does not know how to flash from qemu. I haven't researched this much; it does seem to be possible to write to flash from qemu, but I don't know what is needed in flashrom to enable this.

---

Original post:

I was able to get Heads working pretty well in qemu using a software TPM and USB-forwarded token. Booting, HOTP, TOTP, TPM reset, etc. all work. I think this is a pretty useful way to iterate on Heads, once it is set up it is pretty easy to rebuild and retest.

A remaining limitation is that Heads can't reflash itself from qemu, which means it can't update the GPG keyring. There's now a way to inject a GPG public key at build time to work around this. This is probably solvable, qemu does seem to support writing to flash, I haven't looked into the details of supporting this in flashrom though.

There is no NVRAM currently either, qemu only supports 8 MB of flash space, and the Heads image uses all of it. It is probably possible to extend this in qemu, but I haven't looked into it yet.

Bootstrapping the entire setup from scratch is possible, though there are a number of steps, which I documented in README.md. Some steps could be varied if you already have an OS installed via SeaBIOS, or already have a GPG key to import, but I didn't write about all the variations to try and keep one clear path in the doc.

---

Fixes #354 #516 and #701

[tlaurion]

First things first, welcome @JonathonHall-Purism !

There is no NVRAM currently either, qemu only supports 8 MB of flash space, and the Heads image uses all of it. It is probably possible to extend this in qemu, but I haven't looked into it yet.

It is possible to select another image size ffrom coreboot config files. One of the old reason why it was locked into 8mb was to mimic lower sized boards chips to detect build breakage when including new stuff/migrating to newer Linux versions etc. I do not think this is a problem anymore and could be increased.

There's now a way to inject a GPG public key at build time to work around this. This is probably solvable, qemu does seem to support writing to flash, I haven't looked into the details of supporting this in flashrom though.

Awesome that public key can be included. I tried to find references for qemu internal reflashing but didn't find any hints to accomplish this.

Bootstrapping the entire setup from scratch is possible, though there are a number of steps, which I documented in README.md. Some steps could be varied if you already have an OS installed via SeaBIOS, or already have a GPG key to import, but I didn't write about all the variations to try and keep one clear path in the doc.

Will try to replicate the results in the next coming days and comment back!

[JonathonHall-Purism]

Thanks @tlaurion ! I'm excited to contribute to the project 💯

(re: ROM size) I do not think this is a problem anymore and could be increased.

The problem there isn't the rom size in coreboot - it's that qemu is not able to map more than 8 MB of flash space. We'd have to raise that limit in qemu (which I would guess is probably not too difficult, but devil is in the details I'm sure). Currently if you try to enable NVRAM you get an error like "combined size of system firmware exceeds 8388608 bytes".

Or we could decrease the coreboot ROM size to make room for NVRAM, but there is not much free space right now (I'm showing ~170 KB free). I figure we will probably run into the 8 MB ceiling in qemu sooner or later anyway, but there's definitely more I need to understand about this to have the complete picture.

Will try to replicate the results in the next coming days and comment back!

Amazing, let me know how it goes!

[JonathonHall-Purism]

Improved swtpm/libtpms build instructions for Debian - Tray provided instructions to build deb packages, which plays much more nicely with AppArmor (thanks Tray!)

[JonathonHall-Purism]

Simplified the "virtual USB flash drive" instructions - Daniel found that I had omitted the step to create the partition table between the dd and kpartx, but since Heads can handle a formatted device with no partition table, that simplifies the setup a lot (no more kpartx at all, thanks Daniel!)

[tlaurion]

Seems like this is moving!

Some questions @JonathonHall-Purism :

• Is KVM required (virtio) or can this skip virsh and virt-manager altogether and be ran as a qemu command on make run as it was previously from board config?

• Also, how hard would but be to create swtpm module so that having that make BOARD run build swtpm and dependencies from source?

Asking because running Qubes here and nesting virt is not supported. So basically postponing until I install another OS on a separated disk.

Let me know, otherwise I will test this asap but delaying because I cannot just... Test this on current setup.

[JonathonHall-Purism]

@tlaurion KVM isn't a hard requirement of libvirt - I think the simplest path would be to continue using libvirt/virt-manager/virsh but let it drive a QEMU VM without KVM when KVM isn't available. It looks like just tweaking the domain type in the XML from 'kvm' to 'qemu' does this. I gave this a quick try inside a VM (so Heads was nested), it's slower but still bearable and does seem to work.

I'd like to see if I can have it pick 'kvm' or 'qemu' automatically since the speed difference is pretty noticeable, but I'd like it to be relatively easy to use on Qubes too.

libvirt buys us quite a bit for this setup IMO, so while I'm sure it's possible to build/drive swtpm and qemu manually, I think we will end up reimplementing a lot of libvirt:

• it knows how to set up the swtpm persistent store and remember where it is
• it knows how to start up swtpm and hook up whatever socket it provides to qemu, and it'll shut it down
• we'd have to give more customization params for the user to tell us how to USB-forward the token into the VM, libvirt has options to do this by device ID or port, etc., and virt-manager is an easy config tool
• similar for a persistent disk, you'd have to create it manually and tell us where it is to attach to the VM

I think to make run a relatively-persistent config with the same TPM/disk/token could end up requiring a small boatload of variables 😢

[tlaurion]

I confirm TPM functional, was able to replay instructions to create swtpm and libtpms packages and install, with some comments for review.

Also, from Qubes, I had to reduce memory and change network settings to try to approximate what was under make run, with no perfect result to test deployed xml:

diff --git a/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml b/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
index ce8fa755..15ce22d0 100644
--- a/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
+++ b/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
@@ -1,7 +1,7 @@
-<domain type='kvm'>
+<domain type='qemu'>
   <name>heads-qemu-coreboot-fbwhiptail</name>
-  <memory unit='KiB'>6291456</memory>
-  <currentMemory unit='KiB'>6291456</currentMemory>
+  <memory unit='KiB'>1000000</memory>
+  <currentMemory unit='KiB'>1000000</currentMemory>
   <vcpu placement='static'>2</vcpu>
   <os>
     <type arch='x86_64' machine='q35'>hvm</type>
@@ -14,9 +14,10 @@
   </features>
   <devices>
     <emulator>/usr/bin/qemu-system-x86_64</emulator>
-    <interface type='network'>
-      <source network='default'/>
-      <model type='virtio'/>
+    <interface type="user">
+      <mac address="52:54:00:6f:ab:33"/>
+      <model type="e1000"/>
+      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
     </interface>
     <tpm model='tpm-tis'>
       <backend type='emulator' version='1.2'/>

It is to note that Qubes doesn't pass complete USB controllers, but usb devices to VMs. I was not able to pass Nitrokey Pro to have it seen by qemu. Were your tests with qemu giving any different results @JonathonHall-Purism ?
I was able to add controller, my bad

It is to note that 2715851#diff-d68adaf3548aa4d283e0063fc497bc0bee572e9b64cafec9a15a8bcc428b351cR64-R86 from #893 was approaching a correct make run from qemu, but swtpm and libtpm were not made as modules which made the process complicated (which you resolved per documentation).

Some other insights from Trammel in this commit for good qemu passed arguments: 2438db5, if that is of any help.

Any insights?

[tlaurion]

@JonathonHall-Purism

Some input for qemu/kvm network config

It seems that for e1000e to be functional, we need to specify a user->virtualization mapping otherwise Intel e1000 controller is not found (you can try this through network-init-recovery script from Heads recovery shell)

The e1000e driver is loaded but nothing is mapped under qemu, since qemu is not finding the controller.
To make this work:

diff --git a/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml b/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
index ce8fa755..8d53d461 100644
--- a/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
+++ b/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml

@@ -1,7 +1,7 @@
-<domain type='kvm'>
+<domain type='qemu'>
   <name>heads-qemu-coreboot-fbwhiptail</name>
-  <memory unit='KiB'>6291456</memory>
-  <currentMemory unit='KiB'>6291456</currentMemory>
+  <memory unit='KiB'>1000000</memory>
+  <currentMemory unit='KiB'>1000000</currentMemory>
   <vcpu placement='static'>2</vcpu>
   <os>
     <type arch='x86_64' machine='q35'>hvm</type>
@@ -14,9 +14,9 @@
   </features>
   <devices>
     <emulator>/usr/bin/qemu-system-x86_64</emulator>
-    <interface type='network'>
-      <source network='default'/>
-      <model type='virtio'/>
+    <interface type="user">
+      <model type="e1000"/>
+      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
     </interface>
     <tpm model='tpm-tis'>
       <backend type='emulator' version='1.2'/>

Also, the current coreboot configuration is not activating measured boot at all. Consequently, cbmem -L is not providing TCPA log, and you can also see that PRC2 is not populated either (cat /sys/class/tpm/tpm0/pcrs).

To make measured boot work

diff --git a/config/coreboot-qemu-fbwhiptail.config b/config/coreboot-qemu-fbwhiptail.config
index 5952dad0..0aed2d6d 100644
--- a/config/coreboot-qemu-fbwhiptail.config
+++ b/config/coreboot-qemu-fbwhiptail.config
@@ -10,6 +10,8 @@ CONFIG_PCIEXP_ASPM=y
 CONFIG_PCIEXP_COMMON_CLOCK=y
 CONFIG_UART_PCI_ADDR=0
 CONFIG_DRIVERS_PS2_KEYBOARD=y
+CONFIG_USER_TPM1=y
+CONFIG_TPM_MEASURED_BOOT=y
 CONFIG_DEFAULT_CONSOLE_LOGLEVEL_6=y
 CONFIG_PAYLOAD_LINUX=y
 CONFIG_PAYLOAD_FILE="../../build/qemu-coreboot-fbwhiptail/bzImage"

As you can see, coreboot configuration needs to know which TPM standard to use. (TPM2 would require in the future: CONFIG_USER_TPM2=y and also a change in xml file for virt-manager to deal correctly with TPM implementation as well)

Maybe we should leave current qemu and qemu-fbwhiptail configurations alone and create -tpm1 new coreboot and board configurations? And later on have -tpm2 board additions?

Also, qemu-coreboot-fbwhiptail (which had no tpm and no HOTP support) now requires both.
Consequently, for the sake of this actual PR, I think it would be a good idea to add qemu-coreboot-fbwhiptail-tpm1-hotp and qemu-coreboot-tpm1-hotp, or only qemu-coreboot-fbwhiptail-tpm1-hotp and its associated coreboot configuration file?

Otherwise, non-fbwhiptail board could have -tpm1 but not hotp, but requiring both closes in to only test Nitrokey/Purism Librem Key (HOTP support) which would not cover TOTP base use case.

Thoughts?

[tlaurion]

Also, on qemu (Qubes OS qube) to add USB dongle, user has to:

• pass usb device to qube
• open virt-manager's vm
• click "Show virtual hardware details"
• right click, select add new hardware, go to "USB Host device" and select Nitrokey/Purism device

Then USB Security dongle is accessible.

[tlaurion]

Also note that using virt-manager implies expected peristence of data under /var/lib/libvirt.
This is not the case under Qubes, where only /home /usr/local and bind-dirs have persistence.

[tlaurion]

@JonathonHall-Purism :

Here is the board config and coreboot config I used to successfully use swtpm based on prior @osresearch referred work.

As current, I cannot successfully pass USB thumb drive to qemu, while the passing Nitrokey from host to qube through usb-proxy exposes the device correctly.

Note here addition of qubes_run statement in board config:

diff --git a/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config b/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config
new file mode 100644
index 00000000..9502b25b
--- /dev/null
+++ b/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config
@@ -0,0 +1,113 @@
+# Configuration for building a coreboot ROM that works in
+# the qemu emulator in GUI mode thanks to FBWhiptail
+#
+# TPM can be used with a qemu software TPM (TIS, 1.2).  A Librem Key or
+# Nitrokey Pro can also be used by forwarding the USB device from the host to
+# the VM.
+export CONFIG_COREBOOT=y
+export CONFIG_COREBOOT_VERSION=4.13
+export CONFIG_LINUX_VERSION=4.14.62
+
+CONFIG_COREBOOT_CONFIG=config/coreboot-qemu-fbwhiptail-tpm1-hotp.config
+CONFIG_LINUX_CONFIG=config/linux-qemu.config
+
+ifeq "$(CONFIG_UROOT)" "y"
+CONFIG_BUSYBOX=n
+else
+CONFIG_KEXEC=y
+CONFIG_QRENCODE=y
+CONFIG_TPMTOTP=y
+CONFIG_POPT=y
+CONFIG_FLASHTOOLS=y
+CONFIG_FLASHROM=y
+CONFIG_PCIUTILS=y
+CONFIG_UTIL_LINUX=y
+CONFIG_CRYPTSETUP2=y
+CONFIG_GPG2=y
+CONFIG_LVM2=y
+CONFIG_MBEDTLS=y
+CONFIG_DROPBEAR=y
+CONFIG_MSRTOOLS=y
+CONFIG_HOTPKEY=y
+
+#Uncomment only one of the following block
+#Required for graphical gui-init (FBWhiptail)
+CONFIG_CAIRO=y
+CONFIG_FBWHIPTAIL=y
+#
+#text-based init (generic-init and gui-init)
+#CONFIG_NEWT=y
+#CONFIG_SLANG=y
+
+endif
+
+export CONFIG_LINUX_USB_COMPANION_CONTROLLER=y
+CONFIG_LINUX_USB=y
+CONFIG_LINUX_E1000=y
+
+#Uncomment only one BOOTSCRIPT:
+#Whiptail-based init (text-based or FBWhiptail)
+export CONFIG_BOOTSCRIPT=/bin/gui-init
+#
+#text-based original init:
+#export CONFIG_BOOTSCRIPT=/bin/generic-init
+
+export CONFIG_TPM=y
+
+export CONFIG_BOOT_DEV="/dev/sda1"
+export CONFIG_BOARD_NAME="qemu-coreboot-fbwhiptail-tpm1-hotp"
+
+#borrowed from https://github.com/orangecms/webboot/blob/boot-via-qemu/run-webboot.sh
+TPMDIR=$(build)/$(BOARD)/vtpm
+qubes_run:
+	@mkdir -p "$(TPMDIR)"
+	swtpm socket \
+		--tpmstate dir="$(TPMDIR)" \
+		--flags "startup-clear" \
+		--ctrl type=unixio,path="$(TPMDIR)/sock" &
+	sleep 0.5
+
+	-qemu-system-x86_64 \
+		--machine q35 \
+		-m 1G \
+		--serial /dev/tty \
+		--bios $(build)/$(BOARD)/$(CB_OUTPUT_FILE) \
+		-object rng-random,filename=/dev/urandom,id=rng0 \
+		-device virtio-rng-pci,rng=rng0 \
+		-netdev user,id=u1 -device e1000,netdev=u1 \
+		-chardev socket,id=chrtpm,path="$(TPMDIR)/sock" \
+		-tpmdev emulator,id=tpm0,chardev=chrtpm \
+		-device tpm-tis,tpmdev=tpm0 \
+		-usb -device usb-host,hostbus=1,hostport=1 \
+
+	stty sane
+
+# Provision a persistent VM using this firmware and a software TPM - create the
+# VM, or if it already exists, update the ROM configuration.  The machine can
+# then have a disk attached in order to boot an OS, and a USB token can be
+# forward into the guest for HOTP support.
+#
+# This VM is configured with libvirt rather than invoked ad-hoc, so that
+# libvirt will manage the swtpm invocation and persistent disks can be
+# configured easily.
+
+# Default for virsh is qemu:///session, but default for virt-manager is
+# qemu:///system
+LIBVIRT_CONNECTION ?= qemu:///system
+
+# Use the GPG-injected ROM if a key was given, since we can't reflash a GPG
+# keyring in QEMU.  Otherwise use the plain ROM, some things can still be tested
+# that way without a GPG key.
+ifneq "$(PUBKEY_ASC)" ""
+QEMU_BOOT_ROM := $(build)/$(BOARD)/$(CB_OUTPUT_FILE_GPG_INJ)
+else
+QEMU_BOOT_ROM := $(build)/$(BOARD)/$(CB_OUTPUT_FILE)
+endif
+
+provision: $(QEMU_BOOT_ROM)
+	if ! virsh -c "$(LIBVIRT_CONNECTION)" domuuid heads-qemu-coreboot-fbwhiptail 2>/dev/null; then \
+		virsh -c "$(LIBVIRT_CONNECTION)" define "boards/$(BOARD)/heads-qemu-coreboot-fbwhiptail.xml"; \
+	fi
+	virsh -c "$(LIBVIRT_CONNECTION)" dumpxml heads-qemu-coreboot-fbwhiptail \
+		| xmlstarlet ed --update /domain/os/loader --value "$(QEMU_BOOT_ROM)" \
+		| virsh -c "$(LIBVIRT_CONNECTION)" define /dev/stdin
diff --git a/config/coreboot-qemu-fbwhiptail-tpm1-hotp.config b/config/coreboot-qemu-fbwhiptail-tpm1-hotp.config
new file mode 100644
index 00000000..6bf546f6
--- /dev/null
+++ b/config/coreboot-qemu-fbwhiptail-tpm1-hotp.config
@@ -0,0 +1,18 @@
+# CONFIG_INCLUDE_CONFIG_FILE is not set
+CONFIG_CBFS_SIZE=0x780000
+# CONFIG_POST_IO is not set
+# CONFIG_POST_DEVICE is not set
+CONFIG_BOARD_EMULATION_QEMU_X86_Q35=y
+# CONFIG_CONSOLE_SERIAL is not set
+CONFIG_LINUX_COMMAND_LINE="debug"
+CONFIG_COREBOOT_ROMSIZE_KB_8192=y
+CONFIG_PCIEXP_ASPM=y
+CONFIG_PCIEXP_COMMON_CLOCK=y
+CONFIG_UART_PCI_ADDR=0
+CONFIG_DRIVERS_PS2_KEYBOARD=y
+CONFIG_USER_TPM1=y
+CONFIG_TPM_MEASURED_BOOT=y
+CONFIG_DEFAULT_CONSOLE_LOGLEVEL_6=y
+CONFIG_PAYLOAD_LINUX=y
+CONFIG_PAYLOAD_FILE="../../build/qemu-coreboot-fbwhiptail-tpm1-hotp/bzImage"
+CONFIG_LINUX_INITRD="../../build/qemu-coreboot-fbwhiptail-tpm1-hotp/initrd.cpio.xz" 

Waiting for your comments on:

• code review
• insights on feasibility to pass kvm to qemu (maybe have different configs and board statements would do) and
• fixes/tests of proposed xml network config as opposed to using (non-always) existing default.
• /var non-persistence over Qubes OS unless the qcow disk image is created also under board dir, just like the swtpm state dir is created in board config proposed above (we can take for granted that that directory is under persistent storage).
• Any hint on how to pass usb storage device to qemu directly (see below)?
  Otherwise, I will work on that later on. The final passed qemu (which can boot from USB device) looks like this, and this is the farther I was able to get today in my testings:
  /usr/bin/qemu-system-x86_64 -name guest=heads-qemu-coreboot-fbwhiptail,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-heads-qemu-coreboot-/master-key.aes -blockdev {"driver":"file","filename":"/home/user/heads/build/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail-v0.2.1.bis2-105-g04a46efc-dirty.rom","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -machine pc-q35-5.2,accel=tcg,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,memory-backend=pc.ram -cpu qemu64 -m 977 -object memory-backend-ram,id=pc.ram,size=1024458752 -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid b6731524-d821-4050-8273-2af7913df36d -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=34,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device qemu-xhci,id=usb,bus=pci.1,addr=0x0 -netdev user,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:da:1f:19,bus=pcie.0,addr=0x3 -tpmdev emulator,id=tpm-tpm0,chardev=chrtpm -chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/1-heads-qemu-coreboot--swtpm.sock -device tpm-tis,tpmdev=tpm-tpm0,id=tpm0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device bochs-display,id=video0,vgamem=16384k,bus=pcie.0,addr=0x1 -device usb-host,hostdevice=/dev/bus/usb/001/009,id=hostdev0,bus=usb.0,port=1 -device usb-host,hostdevice=/dev/bus/usb/002/011,id=hostdev1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.5,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

Which seems to map -device usb-host with the host-device directly (still qubes here, so can be generalizable).
Snippet from above:
-device usb-host,hostdevice=/dev/bus/usb/001/009,id=hostdev0,bus=usb.0,port=1 -device usb-host,hostdevice=/dev/bus/usb/002/011,id=hostdev1,bus=usb.0,port=3

[JonathonHall-Purism]

Amazing, thanks for all the testing and input @tlaurion ! Lots of items to address, I'm going to run through this on Qubes myself to get all the feedback integrated.

1. Code review - Will fix those errors, thanks! 🤦
2. TPM/key in qemu - Yeah I agree let's make this a separate board config, I will split it up.
3. /var non-persistence - OK, I see where you're going with the bare-qemu approach, using fixed locations for persistence in the board directory. I did also like the libvirt approach because it offered a lot of flexibility in (re-)configuring the VM, shuffling disks around without reinstalling OSes, etc., but maybe I can do that with some scripts and symlinks instead. I'll see how far I can get with this approach.
4. USB storage to qemu - I'm not sure offhand how to do this, but I will look at what libvirt does, should offer some insight.

I have a handful of other things needing my attention today, but I'll get back to this soon, just know I haven't forgotten 😁

[JonathonHall-Purism]

@tlaurion Reworked this quite a bit - eliminated libvirt, and the whole process works in Qubes now. I rewrote the instructions in the README, I think it's a little bit simpler.

PureOS and Debian live LXDE both work and install. GNOME is brutally slow nested under Qubes but it works. However, Debian netinst and Fedora server won't boot with Bochs video (not specific to Heads, although outside of Heads there is a text-based fallback mode menu). Supporting QXL or virtio in Heads would be nice but this is still plenty to test with.

I really wanted to have the 'run' target depend on the actual ROM, but re-running didn't work - the ROM was always altered on re-run so the TOTP secret could never be unsealed. I didn't look into this much yet but I don't think it needs to block this PR.

There's still a sticky detail that if the qemu invocation fails for any reason, swtpm is left running and must be killed manually. Maybe there's some make-fu to take care of that, but it's not that big an issue and the script is fairly complex already.

[JonathonHall-Purism]

I don't think this is related to this PR, looks like the debian:11 image has been updated compared to the last successful builds. Any ideas offhand @tlaurion ?

configure:6738: checking termio.h usability
configure:6738: /root/project/crossgcc/bin/x86_64-linux-musl-gcc -fdebug-prefix-map=/root/project=heads -gno-record-gcc-switches -D__MUSL__ -isystem /root/project/install/include -L/root/project/install/lib  -c -g -O2 -Wall -Wno-pointer-sign -Wpointer-arith  -Wall conftest.c >&5
conftest.c:63:10: fatal error: termio.h: No such file or directory
 #include <termio.h>
          ^~~~~~~~~~

[tlaurion]

The cache couldn't be restored
https://app.circleci.com/pipelines/github/osresearch/heads/431/workflows/0c946ab1-673e-4512-a877-9f3ed9e417e2/jobs/4076/parallel-runs/0/steps/0-108

@JonathonHall-Purism
Go under circleci project settings, environment variable

And add a new CACHE_VERSION variable so that you start a fresh build without cache, as per CircleCI cache settings:

https://github.com/osresearch/heads/blob/master/.circleci/config.yml#L80

[tlaurion]

It also failed trying to download
https://app.circleci.com/pipelines/github/osresearch/heads/431/workflows/0c946ab1-673e-4512-a877-9f3ed9e417e2/jobs/4077/parallel-runs/0/steps/0-103

The error is on the board build, where all logs are outputted in the logs at next step.

I would simply add environment variable to retry1 as indicated on previous step and retry the build.

[tlaurion]

I just pushed the retry button to see if error is reproducible

[tlaurion]

Hmm first time that I have access denied on cache...

https://app.circleci.com/pipelines/github/osresearch/heads/431/workflows/c97b3e67-f9a3-4f8e-b365-09be60e063c9/jobs/4078/parallel-runs/0/steps/0-108

Will change variable and trigger rebuild

[JonathonHall-Purism]

matt.ucc.asn.au seems to be completely down for dropbear, but there is a mirror: https://mirror.dropbear.nl/mirror/releases/dropbear-2016.74.tar.bz2

[tlaurion]

Prep step running without cache
https://app.circleci.com/pipelines/github/osresearch/heads/431/workflows/57899f73-9d46-4578-8244-e9bb1aada9ba/jobs/4080

Hopefully it will build clean.

I think I see what is happening here.

I would advise everyone at purism to follow their clones from circle ci.

This will prevent the builds to consume all free circleci credits of osresearch account and would build on your own personal accounts.

Also, when build errors like this happens, each user can change their environment variables to start from clean builds.

• Maybe we should document this under heqds-wiki development section

[tlaurion]

matt.ucc.asn.au seems to be completely down for dropbear, but there is a mirror: https://mirror.dropbear.nl/mirror/releases/dropbear-2016.74.tar.bz2

Unfortunately typing from cell phone here away from keyboard. But I'm getting sick of this.

• That would deserve another issue to point all current packages to Debian archive mirrors for all heads packages. Those name resolution issues have been a toll and prevents as well reproducibility of builds for commit over time.

[tlaurion]

@JonathonHall-Purism please rebase on master.

[JonathonHall-Purism]

Thanks, I sent a PR to switch to the mirror for now, I expect this one will still fail until that is merged in. Agree that using the Debian archive would be a good move. I'll check back on these tomorrow.

Thanks for the advice on CircleCI too, I set CACHE_VERSION and configured my fork of heads in CircleCI under my account.

[tlaurion]

@JonathonHall-Purism merged your PR since there is no checksum change implied.

[tlaurion]

You have successfully determined a proper custom partition scheme that permits Heads to boot such install?
(Legacy portions flags and whatnot?)
Yeah I just used the Debian netinst image (which boots now) to install with XFCE desktop and it works, you do have to create an unencrypted /boot manually.

I can add a section about OS support with those notes. This isn't specific to qemu though so I think this should be a separate section in README.md, agree @tlaurion ?

Agreed! Note that everything deeper doc is under heads-wiki (rendered over https://osresearch.net)

I would also suggest editing OP to summarize the final stage of this PR, so that people do not have to read all that was written here.

[JonathonHall-Purism]

Update to docs: linuxboot/heads-wiki#99

Mentioned there that maybe README.md should link to those setup steps instead of duplicating them, do you have a preference @tlaurion ?

[tlaurion]

@JonathonHall-Purism your comment was really pertinent over there.

Maybe instructions should be self-contained in the board config (so under heads) and referred in heads-wiki and under heads README?

So that we do not have to maintain it in 3 places :)

[tlaurion]

@JonathonHall-Purism FYI, this PR was already really useful under #1200 (comment) (quick hack to modify it to run without fbwhiptail and without hotp and killing swtpm after qemu launch to simulate a board a shared board with/without TPM).

[tlaurion]

@JonathonHall-Purism : The following was used in previous tests, which works way better over qemu since in console mode (whiptail, not fbwhiptail).
This would be no-op for you to add, where board config name and content states the differences.

A quick note on killing swtpm (killall swtpm prior of Heads initrd being unpacked) to disable TPM to use alternative code path (NO TPM: Purism mini, Nitrokey mini) worked as intended for #1200 testing.

diff --git a/boards/qemu-coreboot-whiptail-tpm1/qemu-coreboot-whiptail-tpm1.config b/boards/qemu-coreboot-whiptail-tpm1/qemu-coreboot-whiptail-tpm1.config
new file mode 100644
index 00000000..633aa09e
--- /dev/null
+++ b/boards/qemu-coreboot-whiptail-tpm1/qemu-coreboot-whiptail-tpm1.config
@@ -0,0 +1,150 @@
+# Configuration for building a coreboot ROM that works in
+# the qemu emulator in console mode thanks to Whiptail
+#
+# TPM can be used with a qemu software TPM (TIS, 1.2).
+export CONFIG_COREBOOT=y
+export CONFIG_COREBOOT_VERSION=4.13
+export CONFIG_LINUX_VERSION=5.10.5
+
+CONFIG_COREBOOT_CONFIG=config/coreboot-qemu-whiptail-tpm1.config
+CONFIG_LINUX_CONFIG=config/linux-qemu.config
+
+ifeq "$(CONFIG_UROOT)" "y"
+CONFIG_BUSYBOX=n
+else
+CONFIG_KEXEC=y
+CONFIG_QRENCODE=y
+CONFIG_TPMTOTP=y
+CONFIG_POPT=y
+CONFIG_FLASHTOOLS=y
+CONFIG_FLASHROM=y
+CONFIG_PCIUTILS=y
+CONFIG_UTIL_LINUX=y
+CONFIG_CRYPTSETUP2=y
+CONFIG_GPG2=y
+CONFIG_LVM2=y
+CONFIG_MBEDTLS=y
+CONFIG_DROPBEAR=y
+CONFIG_MSRTOOLS=y
+#CONFIG_HOTPKEY=y
+
+#Uncomment only one of the following block
+#Required for graphical gui-init (FBWhiptail)
+#CONFIG_CAIRO=y
+#CONFIG_FBWHIPTAIL=y
+#
+#text-based init (generic-init and gui-init)
+CONFIG_NEWT=y
+CONFIG_SLANG=y
+
+endif
+
+export CONFIG_LINUX_USB_COMPANION_CONTROLLER=y
+CONFIG_LINUX_USB=y
+CONFIG_LINUX_E1000=y
+
+#Uncomment only one BOOTSCRIPT:
+#Whiptail-based init (text-based or FBWhiptail)
+export CONFIG_BOOTSCRIPT=/bin/gui-init
+#
+#text-based original init:
+#export CONFIG_BOOTSCRIPT=/bin/generic-init
+export CONFIG_BOOT_REQ_HASH=n
+export CONFIG_BOOT_REQ_ROLLBACK=n
+export CONFIG_BOOT_KERNEL_ADD="console=ttyS0 console=tty systemd.zram=0"
+export CONFIG_BOOT_KERNEL_REMOVE="quiet rhgb splash"
+
+export CONFIG_TPM=y
+
+export CONFIG_BOOT_DEV="/dev/vda1"
+export CONFIG_BOARD_NAME="qemu-coreboot-fbwhiptail-tpm1-hotp"
+
+# Use the GPG-injected ROM if a key was given, since we can't reflash a GPG
+# keyring in QEMU.  Otherwise use the plain ROM, some things can still be tested
+# that way without a GPG key.
+ifneq "$(PUBKEY_ASC)" ""
+QEMU_BOOT_ROM := $(build)/$(BOARD)/$(CB_OUTPUT_FILE_GPG_INJ)
+else
+QEMU_BOOT_ROM := $(build)/$(BOARD)/$(CB_OUTPUT_FILE)
+endif
+
+#borrowed from https://github.com/orangecms/webboot/blob/boot-via-qemu/run-webboot.sh
+TPMDIR=$(build)/$(BOARD)/vtpm
+$(TPMDIR)/.manufacture:
+	mkdir -p "$(TPMDIR)"
+	swtpm_setup --tpm-state "$(TPMDIR)" --create-platform-cert --lock-nvram
+	touch "$(TPMDIR)/.manufacture"
+ROOT_DISK_IMG=$(build)/$(BOARD)/root.qcow2
+# Default to 20G disk
+QEMU_DISK_SIZE?=20G
+$(ROOT_DISK_IMG):
+	qemu-img create -f qcow2 "$(ROOT_DISK_IMG)" $(QEMU_DISK_SIZE)
+# Remember the amount of memory so it doesn't have to be specified every time.
+# Default to 4G, most bootable OSes are not usable with less.
+QEMU_MEMORY_SIZE?=4G
+MEMORY_SIZE_FILE=$(build)/$(BOARD)/memory
+$(MEMORY_SIZE_FILE):
+	@echo "$(QEMU_MEMORY_SIZE)" >"$(MEMORY_SIZE_FILE)"
+USB_FD_IMG=$(build)/$(BOARD)/usb_fd.raw
+$(USB_FD_IMG):
+	dd if=/dev/zero bs=1M of="$(USB_FD_IMG)" bs=1M count=128
+	# Debian obnoxiously does not include /usr/sbin in PATH for non-root, even
+	# though it is meaningful to use mkfs.vfat (etc.) as non-root
+	MKFS_VFAT=mkfs.vfat; \
+	[ -x /usr/sbin/mkfs.vfat ] && MKFS_VFAT=/usr/sbin/mkfs.vfat; \
+	"$$MKFS_VFAT" "$(USB_FD_IMG)"
+# Pass INSTALL_IMG=<path_to_img.iso> to attach an installer as a USB flash drive instead
+# of the temporary flash drive for exporting GPG keys.
+ifneq "$(INSTALL_IMG)" ""
+QEMU_USB_FD_IMG := $(INSTALL_IMG)
+else
+QEMU_USB_FD_IMG := $(USB_FD_IMG)
+endif
+# To forward a USB token, set USB_TOKEN to one of the following:
+# - NitrokeyPro - forwards a Nitrokey Pro by VID:PID
+# - LibremKey - forwards a Librem Key by VID:PID
+# - <other> - Provide the QEMU usb-host parameters, such as
+#   'hostbus=<#>,hostport=<#>' or 'vendorid=<#>,productid=<#>'
+ifeq "$(USB_TOKEN)" "NitrokeyPro"
+QEMU_USB_TOKEN_DEV := -device usb-host,vendorid=8352,productid=16648
+else ifeq "$(USB_TOKEN)" "NitrokeyStorage"
+QEMU_USB_TOKEN_DEV := -device usb-host,vendorid=8352,productid=16649
+else ifeq "$(USB_TOKEN)" "Nitrokey3NFC"
+QEMU_USB_TOKEN_DEV := -device usb-host,vendorid=8352,productid=17074
+else ifeq "$(USB_TOKEN)" "LibremKey"
+QEMU_USB_TOKEN_DEV := -device usb-host,vendorid=12653,productid=19531
+else ifneq "$(USB_TOKEN)" ""
+QEMU_USB_TOKEN_DEV := -device "usb-host,$(USB_TOKEN)"
+endif
+
+run: $(TPMDIR)/.manufacture $(ROOT_DISK_IMG) $(MEMORY_SIZE_FILE) $(USB_FD_IMG)
+	swtpm socket \
+		--tpmstate dir="$(TPMDIR)" \
+		--flags "startup-clear" \
+		--terminate \
+		--ctrl type=unixio,path="$(TPMDIR)/sock" &
+	sleep 0.5
+
+	-qemu-system-x86_64 -drive file="$(ROOT_DISK_IMG)",if=virtio \
+		--machine q35,accel=kvm:tcg \
+		-rtc base=utc \
+		-smp "$$(nproc)" \
+		-vga virtio \
+		-full-screen \
+		-m "$$(cat "$(MEMORY_SIZE_FILE)")" \
+		-serial stdio \
+		--bios "$(QEMU_BOOT_ROM)" \
+		-object rng-random,filename=/dev/urandom,id=rng0 \
+		-device virtio-rng-pci,rng=rng0 \
+		-netdev user,id=u1 -device e1000,netdev=u1 \
+		-chardev socket,id=chrtpm,path="$(TPMDIR)/sock" \
+		-tpmdev emulator,id=tpm0,chardev=chrtpm \
+		-device tpm-tis,tpmdev=tpm0 \
+		-device qemu-xhci,id=usb \
+		-device usb-tablet \
+		-drive file="$(QEMU_USB_FD_IMG)",if=none,id=usb-fd-drive,format=raw \
+		-device usb-storage,bus=usb.0,drive=usb-fd-drive \
+		$(QEMU_USB_TOKEN_DEV) \
+
+	stty sane
+	@echo
diff --git a/config/coreboot-qemu-whiptail-tpm1.config b/config/coreboot-qemu-whiptail-tpm1.config
new file mode 100644
index 00000000..b6e14479
--- /dev/null
+++ b/config/coreboot-qemu-whiptail-tpm1.config
@@ -0,0 +1,19 @@
+# CONFIG_INCLUDE_CONFIG_FILE is not set
+CONFIG_ONBOARD_VGA_IS_PRIMARY=y
+CONFIG_CBFS_SIZE=0x980000
+# CONFIG_POST_IO is not set
+# CONFIG_POST_DEVICE is not set
+CONFIG_BOARD_EMULATION_QEMU_X86_Q35=y
+# CONFIG_CONSOLE_SERIAL is not set
+CONFIG_LINUX_COMMAND_LINE="debug console=ttyS0,115200 console=tty"
+CONFIG_COREBOOT_ROMSIZE_KB_10240=y
+CONFIG_PCIEXP_ASPM=y
+CONFIG_PCIEXP_COMMON_CLOCK=y
+CONFIG_UART_PCI_ADDR=0
+CONFIG_DRIVERS_PS2_KEYBOARD=y
+CONFIG_USER_TPM1=y
+CONFIG_TPM_MEASURED_BOOT=y
+CONFIG_DEFAULT_CONSOLE_LOGLEVEL_6=y
+CONFIG_PAYLOAD_LINUX=y
+CONFIG_PAYLOAD_FILE="../../build/qemu-coreboot-whiptail-tpm1/bzImage"
+CONFIG_LINUX_INITRD="../../build/qemu-coreboot-whiptail-tpm1/initrd.cpio.xz"

[JonathonHall-Purism]

@tlaurion Added the new config and moved the existing documentation under the board directory, linked from readme and wiki. Ready to merge or is there anything else you can think of?

[tlaurion]

@JonathonHall-Purism Sorry to be picky, but can all your commits be signed?

[JonathonHall-Purism]

Good catch @tlaurion, fixed, thanks!