User reownership of OEM shipped hardware wizard, based on /boot/oem file presence

[tlaurion]

EDIT: I will update this PR once the staging branch is considered stable. Staging builds can be downloaded here.

Straightened #507 replacement.
Depends on #510. Fixes #475.

Modifies this PR (Empty Keyring Detection) to interact directly with /boot/oem file, dropped in place by the OEM once TOTP/HOTP firmware is sealed and /boot files are signed with OEM's owned LibremKey/NitroKey Pro v2. The OEM then ships the devices separately.

As a result, QubesOS or any other Linux distribution can be preinstalled and firmware and boot integrity is attested at first boot by the user received OEM hardware. He then reowns the hardware himself by reencrypting encrypted drive (LUKS container), reowning the TPM, reowning LibremKey/Nitrokey Pro v2, and recreates /boot checksums and sign them with his newly reowned GPG card prior to first booting preinstalled OS.

[tlaurion]

@marmarek @mfc @andrewdavidwong @kylerankin @osresearch @flammit : Here is how the actual reownership looks like from a UX perspective.

The OEM:

• Flashes a clean rom.

  • Generate Heads ROM ( in the case of x230: make BOARD=x230-flash && make BOARD=x230). And flash it externally to hardware.
  • x230: Backup 4mb SPI flash chip, reflash x230-flash Heads rom on it, Backup 8mb SPI flash chip, apply me_cleaner on it and reflash it externally.
  • x230: On first boot of x230-flash Heads, mount-usb && flash.sh /media/coreboot.rom (12mb rom generated with make BOARD=x230)

• Factory resets LibremKey/Nitrokey Pro v2 from proposed option, which inserts his public key and otrust.txt into the rom:
  

• Seals TOTP/HOTP firmware, as requested per UX:
  

• Installs QubesOS, makes sure NitroKey/LibremKey is NOT CONNECTED in second stage of QubesOS installation (else QubesOS detects the LibremKey/Nitrokey Pro v2 as a USB keyboard and refuses to offer isolation of USB devices by creating a sys-usb qube):
  
  
  
  
  
  
  

• Selects a default boot option, create checksum file of /boot files, signs them with GPG card

• Finally, open recovery shell and mount -o remount,rw /boot && touch /boot/oem && mount -o remount,ro /boot && reboot

• Ships Hardware and GPG card seperatly to the user.

Once the user receives his hardware, he plugs the GPG card in the hardware and boots. He gets the following:

• LibremKey/Nitrokey LED should flash green:
  
  
• And then, the hardware reownership begins:
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
• The hardware reownership finishes here. His LibremKey/Nitrokey flashes green:
  

[marmarek]

Is it correct that attacker that intercept both laptop and nitrokey could forge integrity protection? If so, it may be worth adding (optional) extra measurement - for example let the user send their public key and provide back encrypted HOTP/TOTP secret, so it would be possible to verify platform integrity with yet another tool. Since the secret is reset during re-ownership process, it shouldn't be such a big problem that it ever leaved measured environment (of course as long as attacker wouldn't be able to extract it too).

Another thing to verify here - OS (especially unverified one), even when booted with unmodified firmware, should not be able to extract TOTP/HTOP secret. This apply to both RAM (all copies of the secret should be wiped from RAM before booting OS) and also unsealing it again from TPM. One way to achieve the latter would be extending relevant PCR with xen+kernel(+initrd?). Maybe it's already there?

[andrewdavidwong]

Is it correct that attacker that intercept both laptop and nitrokey could forge integrity protection? If so, it may be worth adding (optional) extra measurement - for example let the user send their public key and provide back encrypted HOTP/TOTP secret, so it would be possible to verify platform integrity with yet another tool. Since the secret is reset during re-ownership process, it shouldn't be such a big problem that it ever leaved measured environment (of course as long as attacker wouldn't be able to extract it too).

Another thing to verify here - OS (especially unverified one), even when booted with unmodified firmware, should not be able to extract TOTP/HTOP secret. This apply to both RAM (all copies of the secret should be wiped from RAM before booting OS) and also unsealing it again from TPM. One way to achieve the latter would be extending relevant PCR with xen+kernel(+initrd?). Maybe it's already there?

I really like this suggestion.

Are there step-by-step end user instructions that customers would follow? I'd like to put myself in the shoes of a security-minded potential customer and go through each step of the process to see whether there are any red flags from that point of view. (E.g., "This step leaves me vulnerable to interdiction," or "This step requires unnecessary trust in a third-party." It could be that there are already security measures in place to guard against these things, but potential customers may not be aware of them or understand them.)

[tlaurion]

@marmarek First of all, sorry for the delay answering.

So we talk about a couple of things here that should be addressed separately as necessary.
General information

1. Integrity protection/measure:
   1. TPM firmware integrity measurements and sealing through TPMTOTP
   2. Nitrokey Pro V2/LibremKey firmware integrity measurements and sealing with HOTP on GPG Card
   3. TPM firmware integrity measurements, TPM sealing of LUKS header dump + user defined disk unlock password, stored in LUKS slot 1
2. Secret creation and storage
   1. TPMTOTP secret generation and storage
   2. HOTP secret generation and storage

Is it correct that attacker that intercept both laptop and nitrokey could forge integrity protection? If so, it may be worth adding (optional) extra measurement - for example let the user send their public key and provide back encrypted HOTP/TOTP secret, so it would be possible to verify platform integrity with yet another tool. Since the secret is reset during re-ownership process, it shouldn't be such a big problem that it ever leaved measured environment (of course as long as attacker wouldn't be able to extract it too).

The Firmware integrity/boot configuration cannot be compromised without the user knowing.
The user is provided with TPMTOTP QR Code through user chosen secure communication channel by the OEM, as are GPG card user's and admin passwords, TPM owner's password and disk recovery password associated with cryptsetup slot0 password change after cryptsetup-reencrypt has reencrypted the drive. This TOTP is generated based on these measurements. As a result, the user can validate upon reception that the 6 digits code showed on screen is the on his chosen smartphone OTP application.

So even if both the LibremKey and firmware were compromised (we will get into that later), the user can still attest integrity by the laptop alone and it's TPM.

The HOTP seal is implemented in TPMTOTP and depends on the TPM measurements. On hardware that doesn't provide a TPM (which is not the case here by board configuration) the HOTP measurement is currently based on checksumming an internal SPI dump. To make it short, HOTP is just an easier way of confirming integrity by inserting user's GPG Card and validating green LED flashing instead of manually validating the TOTP code on his phone. Additionally, when the user sets a disk unlock passphrase, TPMTOTP attestation is done without him knowing, since the integrity measurements need to be valid to release the key to be used as an additional initrd file.

As for the possibility of compromising both the firmware and the GPG card, it is near impossible and don't see how it could be possible without user knowing.

The OEM's public GPG key inserted into the ROM is generated on the GPG card and is bound to it for signing operations. That public key fingerprint is shown to the user at each boot. Factory resetting the key would wipe the keys present in the card and reinject a public key different then the one provided by OEM. Any modification to the firmware would change the PCRs measurements and invalidate HOTP/TOTP integrity measurements, and the TOTP and HOTP unsealing would fail. The user has only 3 bad attempts before the card locks user's/admin's individual PIN operations from the card, after which he would have to factory reset the card, which would change inserted user's public key in the rom, invalidate files signature validation and inform the user of compromise.

Another thing to verify here - OS (especially unverified one), even when booted with unmodified firmware, should not be able to extract TOTP/HTOP secret. This apply to both RAM (all copies of the secret should be wiped from RAM before booting OS) and also unsealing it again from TPM. One way to achieve the latter would be extending relevant PCR with xen+kernel(+initrd?). Maybe it's already there?

The sealing of HOTP/TOTPM is an operation that requires to reboot. Every time the user attempts to enter recovery, files are deleted. They could be shredded I guess, since the ramfs is not journalized. You think it would be required?

[tlaurion]

I will update this PR once the staging branch is considered stable.

[tlaurion]

Is it correct that attacker that intercept both laptop and nitrokey could forge integrity protection? If so, it may be worth adding (optional) extra measurement - for example let the user send their public key and provide back encrypted HOTP/TOTP secret, so it would be possible to verify platform integrity with yet another tool. Since the secret is reset during re-ownership process, it shouldn't be such a big problem that it ever leaved measured environment (of course as long as attacker wouldn't be able to extract it too).
Another thing to verify here - OS (especially unverified one), even when booted with unmodified firmware, should not be able to extract TOTP/HTOP secret. This apply to both RAM (all copies of the secret should be wiped from RAM before booting OS) and also unsealing it again from TPM. One way to achieve the latter would be extending relevant PCR with xen+kernel(+initrd?). Maybe it's already there?

I really like this suggestion.

Are there step-by-step end user instructions that customers would follow? I'd like to put myself in the shoes of a security-minded potential customer and go through each step of the process to see whether there are any red flags from that point of view. (E.g., "This step leaves me vulnerable to interdiction," or "This step requires unnecessary trust in a third-party." It could be that there are already security measures in place to guard against these things, but potential customers may not be aware of them or understand them.)

@andrewdavidwong : The user is guided through a wizard for reownership. I could definitely do screenshots in a guide if that is considered required.

[marmarek]

The user is provided with TPMTOTP QR Code through user chosen secure communication channel by the OEM, as are GPG card user's and admin passwords

Ok, this was the part I missed. Looks good.

The sealing of HOTP/TOTPM is an operation that requires to reboot. Every time the user attempts to enter recovery, files are deleted. They could be shredded I guess, since the ramfs is not journalized. You think it would be required?

Maybe, depending how ramfs (tmpfs?) implements file removal, but I guess the actual content stays in RAM, until used for something else.

And the same before booting the OS (especially the one from USB).

[andrewdavidwong]

Is it correct that attacker that intercept both laptop and nitrokey could forge integrity protection? If so, it may be worth adding (optional) extra measurement - for example let the user send their public key and provide back encrypted HOTP/TOTP secret, so it would be possible to verify platform integrity with yet another tool. Since the secret is reset during re-ownership process, it shouldn't be such a big problem that it ever leaved measured environment (of course as long as attacker wouldn't be able to extract it too).
Another thing to verify here - OS (especially unverified one), even when booted with unmodified firmware, should not be able to extract TOTP/HTOP secret. This apply to both RAM (all copies of the secret should be wiped from RAM before booting OS) and also unsealing it again from TPM. One way to achieve the latter would be extending relevant PCR with xen+kernel(+initrd?). Maybe it's already there?

I really like this suggestion.
Are there step-by-step end user instructions that customers would follow? I'd like to put myself in the shoes of a security-minded potential customer and go through each step of the process to see whether there are any red flags from that point of view. (E.g., "This step leaves me vulnerable to interdiction," or "This step requires unnecessary trust in a third-party." It could be that there are already security measures in place to guard against these things, but potential customers may not be aware of them or understand them.)

@andrewdavidwong : The user is guided through a wizard for reownership. I could definitely do screenshots in a guide if that is considered required.

As a potential customer, I would consider it required. I think that most customers will want to know what to expect and to understand what they're actually buying.

I think that the guide should begin when the user actually places the order or, at least, before the package is received. It would be nice to have some assurance that the entire package wasn't replaced with a malicious one, for example.

[tlaurion]

The user is provided with TPMTOTP QR Code through user chosen secure communication channel by the OEM, as are GPG card user's and admin passwords

Ok, this was the part I missed. Looks good.

The sealing of HOTP/TOTPM is an operation that requires to reboot. Every time the user attempts to enter recovery, files are deleted. They could be shredded I guess, since the ramfs is not journalized. You think it would be required?

Maybe, depending how ramfs (tmpfs?) implements file removal, but I guess the actual content stays in RAM, until used for something else.

@marmarek : From what I get, rootfs, is an instance of tmpfs by default:

What is rootfs?

Rootfs is a special instance of ramfs (or tmpfs, if that's enabled), which is
always present in 2.6 systems. You can't unmount rootfs for approximately the
same reason you can't kill the init process; rather than having special code
to check for and handle an empty list, it's smaller and simpler for the kernel
to just make sure certain lists can't become empty.

Most systems just mount another filesystem over rootfs and ignore it. The
amount of space an empty instance of ramfs takes up is tiny.

If CONFIG_TMPFS is enabled, rootfs will use tmpfs instead of ramfs by
default. To force ramfs, add "rootfstype=ramfs" to the kernel command
line.

So in our case, CONFIG_TMPFS=y is the default, while not visible in savedefconfig, since implicit.

mount
rootfs on / type roots (rw,size=4057276k,nr_inodes=1014319)

So the secrets of concern are the one happening when sealing/unsealing:

user@build-x230-coreboot-48:~/heads$ grep -Rn "/tmp/secret" initrd/
initrd/etc/functions:18:	rm -rf /tmp/secret
initrd/etc/functions:19:	mkdir -p /tmp/secret
grep: initrd/etc/mtab: No such file or directory
Binary file initrd/etc/.functions.swp matches
initrd/bin/kexec-seal-key:9:KEY_FILE="/tmp/secret/secret.key"
initrd/bin/kexec-seal-key:10:TPM_SEALED="/tmp/secret/secret.sealed"
initrd/bin/kexec-seal-key:11:RECOVERY_KEY="/tmp/secret/recovery.key"
initrd/bin/kexec-insert-key:36:INITRD_DIR=/tmp/secret/initrd
initrd/bin/kexec-insert-key:37:SECRET_CPIO=/tmp/secret/initrd.cpio
initrd/bin/unseal-hotp:6:HOTP_SEALED="/tmp/secret/hotp.sealed"
initrd/bin/unseal-hotp:7:HOTP_SECRET="/tmp/secret/hotp.key"
initrd/bin/unseal-totp:6:TOTP_SEALED="/tmp/secret/totp.sealed"
initrd/bin/unseal-totp:7:TOTP_SECRET="/tmp/secret/totp.key"
initrd/bin/kexec-select-boot:305:			-a "$add" -r "$remove" -o "/tmp/secret/initrd.cpio" \
initrd/bin/kexec-unseal-key:11:mkdir -p /tmp/secret
initrd/bin/kexec-unseal-key:13:sealed_file="/tmp/secret/sealed.key"
initrd/bin/kexec-unseal-key:17:	key_file="/tmp/secret/secret.key"
initrd/bin/kexec-unseal-key:41:		rm -f /tmp/secret/sealed || true
initrd/bin/seal-totp:17:TOTP_SECRET="/tmp/secret/totp.key"
initrd/bin/seal-totp:18:TOTP_SEALED="/tmp/secret/totp.sealed"
initrd/bin/seal-libremkey:6:HOTP_SEALED="/tmp/secret/hotp.sealed"
initrd/bin/seal-libremkey:7:HOTP_SECRET="/tmp/secret/hotp.key"
initrd/init:30:mkdir -p /tmp/secret

Since we use tmpfs, it seems that using shred would do the trick, and is compiled in busybox.

So basically, it would be to replace rm by shred here:

user@build-x230-coreboot-48:~/heads$ grep -R "/tmp/secret" initrd/|awk -F ":" {'print $1'}|sort|uniq|while read file; do grep -Hn -e "rm " $file 2>&1|grep -i secret;done 
grep: initrd/etc/mtab: No such file or directory
initrd/bin/kexec-unseal-key:41:		rm -f /tmp/secret/sealed || true
initrd/bin/seal-libremkey:33:rm -f "$HOTP_SECRET"
initrd/bin/seal-totp:46:	rm -f "$TOTP_SECRET"
initrd/bin/unseal-hotp:54:	rm -f "$HOTP_SECRET"
initrd/bin/unseal-hotp:58:rm -f "$HOTP_SECRET"
initrd/bin/unseal-totp:24:	rm -f "$TOTP_SECRET"
initrd/bin/unseal-totp:28:rm -f "$TOTP_SECRET"
initrd/etc/functions:18:	rm -rf /tmp/secret

And the same before booting the OS (especially the one from USB).

@marmarek : shredding instead of removing secret related files would address your concerns?
Fixed in #526

[stacktrust]

(design considerations for future work on platform security lifecycle management, please let me know if there's a better place to post this)

There are supply chain management and support scenarios where an enterprise customer with geographically distributed offices would like new devices to ship from the OEM to a local support consultant that is trusted to install or replace a device at their local office. This reduces the need for IT staff (who may not exist at a small office) to provision devices. It also provides a business model for local consultants to develop expertise in platform security for customized enterprise workflows. Topics to consider:

1. OEMs can ship multiple images (e.g. by geo) and versions. Could there be a public data source which matches image identity with Bill of Materials, including software components that may have been reproducibly built? This would allow customers to privately or publicly share risk and threat data associated with each image.

2. An enterprise could require platform image customizations, e.g. configs or mgmt VM changes to be made by their trusted consultant. When a new device is connected to the local office of an enterprise network for provisioning, how can the enterprise prove that the base image originated from the OEM, that the management VM was blessed by the enterprise and the device was customized by their trusted consultant?

3. During on-site device provisioning, how are device identity keys generated and installed into the device's management VM, to migrate admin authority from the OEM or support consultant to the enterprise's management server?

4. When a device is in service and the customer requires support, can they place an L1 support call to their trusted local consultant, who is granted temporary device access (via enterprise management server) for troubleshooting? When purchasing their L1 support contract from the local consultant, can they pay an additional fee for L2 support from one or more software vendors which produced software components (e.g. VMs) in the device image? This enables their trusted local consultant to work directly with software vendors to address local device issues, improving future software versions to address operational support requirements.

5. When a device is taken out of service, how does the enterprise management server, local IT admin, or trusted support consultant ensure that device keys, config and data are safely removed before the device is recycled or resold? Key ownership lifecycle has been a topic of discussion at OCP (Open Compute Platform) in the context of hardware and firmware root of trust, to enable OCP servers from hyperscale cloud providers to be resold on the secondary market, rather than incurring the environmental cost of device destruction.

[tlaurion]

(design considerations for future work on platform security lifecycle management, please let me know if there's a better place to post this)

There are supply chain management and support scenarios where an enterprise customer with geographically distributed offices would like new devices to ship from the OEM to a local support consultant that is trusted to install or replace a device at their local office. This reduces the need for IT staff (who may not exist at a small office) to provision devices. It also provides a business model for local consultants to develop expertise in platform security for customized enterprise workflows. Topics to consider:

1. OEMs can ship multiple images (e.g. by geo) and versions.  Could there be a public data source which matches image identity with Bill of Materials, including software components that may have been reproducibly built?  This would allow customers to privately or publicly share risk and threat data associated with each image.

2. An enterprise could require platform image customizations, e.g. configs or mgmt VM changes to be made by their trusted consultant.  When a new device is connected to the local office of an enterprise network for provisioning, how can the enterprise prove that the base image originated from the OEM, that the management VM was blessed by the enterprise and the device was customized by their trusted consultant?

3. During on-site device provisioning, how are device identity keys generated and installed into the device's management VM, to migrate admin authority from the OEM or support consultant to the enterprise's management server?

4. When a device is in service and the customer requires support, can they place an L1 support call to their trusted local consultant, who is granted temporary device access (via enterprise management server) for troubleshooting?  When purchasing their L1 support contract from the local consultant, can they pay an additional fee for L2 support from one or more software vendors which produced software components (e.g. VMs) in the device image?  This enables their trusted local consultant to work directly with software vendors to address local device issues, improving future software versions to address operational support requirements.

5. When a device is taken out of service, how does the enterprise management server, local IT admin, or trusted support consultant ensure that device keys, config and data are safely removed before the device is recycled or resold?  Key ownership lifecycle has been a topic of discussion at OCP (Open Compute Platform) in the context of hardware and firmware root of trust, to enable OCP servers from hyperscale cloud providers to be resold on the secondary market, rather than incurring the environmental cost of device destruction.

@stacktrust Those are all really interesting questions that all seem more linked to QubesOS then linked to this particular pull request.

One thing I tried to push to address most of your points is to implement an additional remotely manageable AdminVM.. The idea behind this is that an additional on-demand remotely manageable AdminVM would have permissions over a really limited and precise scope (deploy additional templates and salt recipes to create and manage additional qubes, that would solely depend on deployed templates, not the one deployed and managed through dom0, while dom0 could still have control over templates and qubes deployed by additional AdminVM).

It's not possible right now, so the remote manageability that would be possible right would be to give dom0 access from an onion hidden service, with all dom0 access that comes with it; requiring a total trust to a 3rd party from the user. That is not really seducing to me.

You are more then welcome to open that ticket on QubesOS github issues tracker

[stacktrust]

One thing I tried to push to address most of your points is to implement an additional remotely manageable AdminVM.. The idea behind this is that an AdminVM would have permissions over a really limited and precise scope (deploy additional templates and create and manage additional qubes that solely depend on those templates, not the one deployed and managed by dom0).

It's not possible right now, so the remote manageability would give dom0 access and total trust from the user to a 3rd party.

@tlaurion Thanks for the helpful pointers. We did some preliminary work in http://OpenXT.org to support multiple admin VMs: one per "virtual platform" = group of VMs and related policy, with a dedicated VPN or network path to the remote mgmt server. E.g. OpenXT has a file transfer mechanism (https://github.com/OpenXT/icbinn which uses Xen v4v/Argo) to allow an admin VM to deploy multi-gigabyte virtual disks to dom0. The access control pieces are not yet implemented to isolate admin VMs. I'll look at SaltStack and QubesOS admin VMs to see if there is overlap in goals/approach.

[stacktrust]

with all dom0 access that comes with it; requiring a total trust to a 3rd party from the user.

(editorial tangent ahead ... )

QubesOS, XenClient, Moka5, Virtual Computer, OpenXT, Bromium, Hysolate and Windows Sandbox enabled multi-VM endpoints. The next step is multi-MDM (mobile device management).

Microsoft 365 is moving towards a single subscription price that covers the device hardware lease, Windows license, Office license and security updates. Microsoft Azure Sphere IoT device purchase price includes a license to 10 years of Linux security updates, i.e. remote OS management even if the device/app vendor goes out of business. This mgmt isolation was baked into Sphere silicon, OS and app packaging design for IoT/edge.

The boundaries between OEM firmware, OEM factory image, OS vendor updates, OS-mandated app store and owner-managed enterprise apps are evolving. Users are one remote update away from a workflow-impacting (feature, security) change. As supply chain complexity increases, these remote parties will need to be isolated, granted minimal permission, and monitored by the owners of endpoint devices.

[tlaurion]

Will make a PR including just the changes related to the reownership process and force push.

[tlaurion]

replaced by #551