Wiki needs improvements

[ThatLurker]

The entire install section should be redone as its really hard to follow and understand.

we could "merge" the content from the current guide with this as base https://github.com/mfc/flashing-docs/blob/master/walkthrough%20for%20flashing%20heads%20on%20an%20x230.md

[tlaurion]

@pahakalle : This guide is unfortunately obsolete.

Can you clarify which section was harder to understand?

If there is one guide I would merge, it would be the one from @merge and promote a simple external programmer for users to use and order which, with flashrom compiled can be used from a secondary computer with:
sudo ~/flashrom/flashrom --programmer ch341a_spi -w ~/heads/build/x230-flash/x230-flash.rom -c MX25L3273E

[ThatLurker]

The instructions mainly line 35 (Ok, now comes the time to write the 4MB...) are a bit hard to follow/read beacause its just a lot of text.

The guide merge has on skulls is a actually really good and with small modifications could just replace the one here

[GittCatt]

Well, I think the wiki still needs improvements, but for me the worst part is configuring TPM — there just isn't anything useful there. Are the appropriate sections from the outdated guide quoted by @pahakalle still applicable, maybe?

[tlaurion]

@Freewolny : using a board config enforcing gui-init instead of generic-gui is not covered at all by the current wiki pages, which still covers everything from the command line and per generic-init usage, and where most command line commands were replaced by simpler ones or by the GUI which manages those tasks easily and manage error situations automatically.

(see x230 board config or x230-hotp-verification, which later requires a Nitrokey Storage/Nitrokey Pro v2/ Librem Key to enforce HOTP verification ).

Following last link: screenshots, instructions etc should take into consideration whiptail usage that is applicable with/without beautiful graphic enforced by FBWhiptail (which is unusable in remote terminal).

For example, the kgpe-d16 PR enforces gui-init without fbwhiptail for servers while enforcing graphical interface with FBWhiptail for the workstation. So that even on compatible server platforms, gui-init can be used instead of currently documented generic-init under wiki, which became kinda deprecated.

Meanwhile this beginner's page was created, which doesn't cover configuration and Heads daily usage.

Any help welcome redoing the wiki pages.... Is there volunteers?

[GittCatt]

Well, when I manage to do everything myself – reflashing the boards in X230, installing Heads and so on, I'll gladly edit the wiki to make it work for complete newbies like me… but I have a feeling that before this happens, I will need a ton of support myself, unfortunately.

[tlaurion]

@Freewolny please read last link of my previous reply. Your concerns are answered there.

[GittCatt]

One more question – is it currently possible to have an additional keyfile for LUKS encryption stored in TPM?

[tlaurion]

@Freewolny yes. Its called a Disk Unlock Key and is enforced by setting a boot default under Heads on all boards having a TPM, but Purism ones, by default.

TPM NVRAM space releases the key on good Disk Unlock Key passphrase if LUKS header and firmware measurements are valid.

When setting a new boot default, the Disk Revovery Key (key slot 0) is required to be typed to renew/change Disk Unlock Key passphrase (slot 1).

[GittCatt]

But from what I understand, in the solution described by you, the TPM is in possession of the whole decryption key, but releases it only after I authenticate myself to it. So if there was a way to extract the key from the TPM, the attacker would only need my laptop and would have the whole disk decrypted. Admittedly, if the attacker is able to do it, he can probably also employ an evil maid attack and sniff my decryption password, but it's still more effort, as I see it.

I'm looking more for a solution described here:

If an attacker is able to capture the user's keystrokes (hidden camera, electromagnetic leaks), the attacker doesn't need Evil Maid attack anymore, and so doesn't need to bother with compromising the system boot anymore. This is because the attacker can just sniff the disk decryption password, and then steal the laptop and will get full access to all user data.

In order to prevent such a “keystroke sniffing” attack, one can use an additional sealed secret on the Anti Evil Maid stick that would be used as a keyfile for LUKS (in addition to passphrase). In this case the knowledge of the sniffed LUKS passphrase would not be enough for the attacker to decrypt the disk. This has not been implemented, although would be a simple modification to dracut-antievilmaid module. If you decided to use this approach, don't forget to also create a backup passphrase that doesn't need a keyfile, so that you don't lock yourself from access to your data in case you lose your stick, or upgrade your BIOS, or something! You have been warned, anyway.

[tlaurion]

The threat model adressed currently through Heads is the evil maid able to capture the passphrase typed at boot and expecting to be able to clone the disk while owner is away and be able to decrypt the content at their leisure.

What you are looking for is some kind of 2fa or more specifically the part where you need an external device (something you own) to be able to boot the device.

This is not currently implemented, while when QubesOS will upgrade their Dom0, changes having been merged into cryptsetup and some additional changes inside of Heads could permit what you are looking for. Another option would be to have to have the USB security dongle combined and make the user sign with the dongle prior of permitting boot.

Those implementations have risks themselves.

1. If the user looses his external device (boot device/USB security dongle), he can't boot his system anymore

The mitigation are to:

1. Never leave the computer unattended after having unlocked the computer when roaming in unknown space. Or set a new boot default as often as desired when having doubts of having been eavesdrop. The Disk Recovery Key passphrase should only be typed in safe space (slot 0) while the Disk Unlock Key, released by the TPM, should be changed following your threat model.
2. Have your hard drive separated from your computer when roaming.

[tlaurion]

@Freewolny : basically, you are talking about something like this https://docs.puri.sm/PureBoot/LibremKeyLUKS.html

If it is the case, please open an issue under Heads, not heads-wiki.

[Thrilleratplay]

I recently built and installed heads for the first time. Had I not been familiar with coreboot, I would have been very confused by the wiki. Some details are glossed over, some are found on other pages and others are duplicated in different sections, some are wrong (x220 does not have a 4mb chip, only an 8mb). With linuxboot/heads#703, portions of the wiki could theoretically be simplified. In the meantime, I attempted to unify the documentation but am in over my head.....over my heads?....nope, head, no puns here. I am too new to know where the project has been or where it is going nor arrogant enough to start rewriting large portions of the wiki.

This is what I came up with when trying to reorganize it. Ignore the theme; I repurposed a flatdoc theme made a few years ago. I wanted to focus on the content and not waste time setting up Jekyll. Also, it really should not be one large page, but sections should be consolidated and easily navigable.

A few things to note:

• The Heads FAQ page was copied to the wiki instead of link to an external page. Those not familiar with project and osresearch my not understand why they are being linked to a page on trmm.net. It is a little difficult to preach security when sending potential users to seemingly unrelated sites.
• Created a list of the devices that are supported, should they link to the product pages? It makes sense for Purism products but does it make sense for old Lenovo laptops?
• The build guide gives vague suggestions on the libraries required to build but these will different from platform to platform. What about creating an official docker image, similar to coreboot, for building Heads on? I modified my coreboot-builder-scripts for Heads but have not published them yet (need more testing).
• There is a lot of overlap between this and Skull's flashing guide. Perhaps @merge is open to the idea of creating a unified set of documentation for flashing that can be project agnostic.
• I have been planning on creating abstract scripts to automate certain tasks like creating backups of the original rom via flashrom by running it in a loop until hash matched set is created and rename with a timestamp or flash scripts that create backups first with unique names in case an update goes wrong. Little helper scripts may simplify the wiki even more.

Again, I am new to this project so I may be way off base or out of line with some of these suggests. If so, just mock me and tell me off.

[tlaurion]

@Thrilleratplay : I love the live version sitting at https://thrilleratplay.github.io/heads-wiki/ Consequently of #44. I would be agreeing with the reordering and removing trmm.net links, while most of the screen captures are linked to generic-init instead of #40, with some miseading information here and and there (Xen is not included in Heads since a while, there is no Read only FS under QubesOS, x220 flash is not 4mb... All those should be removed. Generic user guide should be taken out of https://docs.puri.sm/PureBoot.html (CC by SA) and manually republished upstream.

• The Heads FAQ page was copied to the wiki instead of link to an external page. Those not familiar with project and osresearch my not understand why they are being linked to a page on trmm.net. It is a little difficult to preach security when sending potential users to seemingly unrelated sites.

Agreed. Cannot take this decision myself though. @osresearch : docs should be self contained.

• Created a list of the devices that are supported, should they link to the product pages? It makes sense for Purism products but does it make sense for old Lenovo laptops?

Coreboot pages should be referred instead?

• The build guide gives vague suggestions on the libraries required to build but these will different from platform to platform. What about creating an official docker image, similar to coreboot, for building Heads on? I modified my coreboot-builder-scripts for Heads but have not published them yet (need more testing).

The point here would then be: how to recreate those docker images.
I modified the instructions to point to CircleCI and GitlabCI instructions recently. I'm not convinced that a docker image would be the solution to reproducibility, and would lead to unmaintained docker images too. I'm ready to debate the idea though, since maintaining CIs is actually not a so easy task which I underestimated.

• There is a lot of overlap between this and Skull's flashing guide. Perhaps @merge is open to the idea of creating a unified set of documentation for flashing that can be project agnostic.

Agreed. That was a really long debate. The solution proposed is linuxboot/heads#571
Where, as of right now, each commit on Heads github is linked to roms produced on CircleCI (with issue linuxboot/heads#768) with reused cache.

• I have been planning on creating abstract scripts to automate certain tasks like creating backups of the original rom via flashrom by running it in a loop until hash matched set is created and rename with a timestamp or flash scripts that create backups first with unique names in case an update goes wrong. Little helper scripts may simplify the wiki even more.

-v does exactly that. Looping it is not so good idea since if backup and verify fails, that would mean the clip needs to be manually replaced.

[Thrilleratplay]

@tlaurion

• Created a list of the devices that are supported, should they link to the product pages? It makes sense for Purism products but does it make sense for old Lenovo laptops?
  Coreboot pages should be referred instead?

That is a good option for the Lenovo's. I am not sure if that would be confusing to new users, * shrugs * probably could be fixed with it explicit saying it is going to the coreboot docs.

Looping it is not so good idea since if backup and verify fails, that would mean the clip needs to be manually replaced.

Not necessarily, the Raspberry Pi bus speed can introduce enough variance to alter a checksum. I ran into this a few times before switching to a Ch341A. My issue with -v is it requires specifying a file, if a file is not specified, it would be nice if it would read a second time, create a temporary file to verify against then remove it if they match. Originally, I thought the specific chips Lenovo used was buried in the official BIOS upgrade software; I found it then forgot where I saw it but found it again and posted it [heads #716](in linuxboot/heads#716). If the specific chips where known, it could be hard coded in said script. Sadly, this isn't the case. Oh well.

The point here would then be: how to recreate those docker images.

It depends on how you would want to approach it.

1. Unless I am mistaken, the goal of reproducible builds is to create the same file regardless of environment. An official Docker image could be similar to coreboot-sdk where the docker environment is used in the reproducible build CI; moving the package dependency maintenance to the container. If published on docker hub, a user could simply pull the image instead of trying to figure out all of the individual packages build packages for their flavor of Linux. This does not need to be limited to a single distro, something similar to this could be heads-debian-sdk and a similar one could be created using Fedora as a base.
2. A Docker image can be reproduced using the same method that verified Docker images are recreated, building from a SHA256 named compressed file containing the image. Here is an example of one of Debian's. Theoretically, these could be created with the cross compiler already built, making testing and local builds much faster.

EDIT: Of course a few hours after I write this I run into an issue. The head-builder-scripts currently work with x230 and qemu-coreboot but there is an issue with the version of gawk when trying to build qemu-linuxboot. This may actually be another use case for using docker images as gawk is a fairly standard utility in Linux and is a specific version is needed.

I noticed the open issues regarding CircleCi builds, using Guix and a few other build related things. If docker images were to compartmentalize build environment dependencies and referenced in the documentation, it may save a lot of headaches trying to help debug different issues across an infinite possible of users environments. It may also make it worse, I am not familiar enough the range of issues that occur.

As for shared flashing documentation across different projects. That may just be a pipe dream. It is difficult enough with different devices and programmers but the same device can have different chips and there are so many variations of CH341A programmers as well as different RPi and BusPirate versions. This is bad enough when dealing with just answering questions of coreboot users on Reddit, then trying to mix this across with different projects whose user backgrounds and technical experience will vary. Just to add more wrenches, how about translations? This is an issue that should be addressed but is beyond the scope of this issue.

[tlaurion]

@thrill

The build guide gives vague suggestions on the libraries required to build but these will different from platform to platform. What about creating an official docker image, similar to coreboot, for building Heads on? I modified my coreboot-builder-scripts for Heads but have not published them yet (need more testing).

Interesting. Looking into this to troubleshoot linuxboot/heads#608

Edit: refacing the problem of having docker inside of QubesOS. digging it as a first step, will update later.
edit 2: found that I wrote something in the past. replaying https://gist.github.com/tlaurion/9113983bbdead492735c8438cd14d6cd/48a9ce8c536d14d1bf9578ce9055581a9c46b851

[Thrilleratplay]

@tlaurion I knew I was forgetting to do something today. I made an edit in the comment above regarding gawk version issues, it likely won't impact the debugging you are doing but did want to call it out.

[tlaurion]

@Thrilleratplay
linuxboot/heads#811 was merged no?

Docker inside of QubesOS instructions updated https://gist.github.com/tlaurion/9113983bbdead492735c8438cd14d6cd

Running your script
user@coreboot-native:~/coreboot-builder-scripts$ ./build.sh -t 4.12 x230

[Thrilleratplay]

@tlaurion Yeah, I assumed linuxboot/heads#811 was the cause. I need to take steps to try to remove gawk from the heads builder docker image. It is only impacting the qemu-linuxboot build. I will try to take a look at it later but also need to clear off space, I didn't keep an eye on the space all of the different heads builds, among other projects, were using and ran out of space on my laptop. I am running Arch on my coreboot x230 and forgot about docker having issues on Qubes. That would be a good item to have in the wiki as well.

[tlaurion]

@Thrilleratplay

about docker having issues on Qubes

https://gist.github.com/tlaurion/9113983bbdead492735c8438cd14d6cd just reproduced and working.

(still downloading coreboot/coreboot-sdk:1.53' locally which seems to be heavy for my damn slow internet link)

[jtmoree-github-com]

I am reworking the Wiki pages to move OS specifics into subfolders, add recovery shell page, and a few other cleanups. Though I have a first pass (https://github.com/jtmoree-github-com/heads-wiki/tree/docs-refresh) in a fork I think there are syntax problems. I'm attempting to setup the environment to test locally based on this page: https://docs.github.com/en/github/working-with-github-pages/testing-your-github-pages-site-locally-with-jekyll

Am I overthinking this? Is there an easier way to test the docs?

[Thrilleratplay]

@jtmoree-github-com http://osresearch.net/Contributing-to-Heads-wiki/

[tlaurion]

I think there should be maybe more time invested into documenting how to dig code, then to write doc for code that is still changing.

Not sure how to do that.

More comment in code instead of external documentation seems more interesting to me. The maintainership problem once again. We see it already, the documentation is way behind the code base as of now. Way behind. And duplicating documentation out of code actually is the cause of that. Lazyness of reading code is one thing, having it too cryptic is another. But my recommendation seeing the new flow of questions coming in for the Heads project is that maybe a general flow of operations in flow chart should be created (more stable) with clickable links into code, where code is more commented and external documentation is maintained to be as minimalist as possible. Otherwise, I bang my head on the walls everytime I surf around and read comments on Reddit and elsewhere, pointing to old docs, bad understanding of what is happening, simply, in my opinion, because of lack of actual code documentation, and outdated external documentation.

@Thrilleratplay ?

[tlaurion]

Silly idea here, but:

• screenshots of menu, where clicking on it would point to commit version of that code in blame mode would point to where the code lies. And then code should document what happens there. High level, and where functions being external (gui_functions, functions, mount_usb not being mount-usb) should be clarified or simply recentralized for easier comprehension.

What you think? User/Dev/documentarist perspectives welcome.

[GittCatt]

That is an absolutely awesome idea (as a user who might be able to dig into code to some extent, but on a day-to-day basis codes something completely different). If the code would really be well documented, that could be a great help in order to grasp a better understanding of some of the internals. pt., 29 sty 2021, 17:44 użytkownik tlaurion <notifications@github.com> napisał:

…

Silly idea here, but: - screenshots of menu, where clicking on it would point to commit version of that code in blame mode would point to where the code lies. And then code should document what happens there. High level, and where functions being external (gui_functions, functions, mount_usb not being mount-usb) should be clarified or simply recentralized for easier comprehension. What you think? User/Dev/documentarist perspectives welcome. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFKALGLREWEXXFJ4ANTSQTS4LQXBANCNFSM4GRHYM3A> .

[Thrilleratplay]

@tlaurion I am not sure myself. This is part of the reason I made mostly structural changes to the wiki. I started to fall into the trap I think @jtmoree-github-com is approaching.

This is personal perspective, Heads is tantalizing to people worried about security but usually attract those who think naive about encryption, security and open source development. The project itself is brilliant but fragmented. There is a lot of hostility from newbies who think this will be a drop in upgrade and are quickly overwhelmed with the complexity. The poor documentation is usually the biggest complaint.

The idiom "those who can't do, teach", something I never understood until college, becomes the catch 22 with Heads documentation. If you don't know enough about Heads, you start to dive through the code to understand it, becoming to familiar with the code, it is difficult to explain it in a high enough level for those who are brand new can understand and abstract enough where it is not immediately obsolete. This is the trap I feel into, I became so familiar with the code, the wiki makes no sense to me.

A few ideas:

• I need to get back to Heads #885. initrd contains a hodgepodge of different scripts with inconsistent styles and, where there is shared code, the usage is not well documented. I believe this is the source of a lot of bugs and grief whenever an update is made. My goal is to create a shared, well documented, fully unit tested framework of functions similar to initrd/etc/functions and initrd/etc/gui_functions but broken down into manageable and identifiable .sh files (a basic example is in the ticket). Any Heads related command/script would be stripped down to only a call to some of these framework functions and a usage function. So, for example, if kexec-boot were executed in the rescue shell, it would print a description of what it does and all of the available options, the script itself would have no logic (other than maybe arg parsing and basic error checking), and simply calls the function that assembles kexec execution. Documentation can be generated by executing the usages of each of these end user facing scripts.
• illustrations to explain Heads structure #64 On the front page of the wiki, there should be diagram explaining the ring of trust the exists between the Heads Rom, TPM and the physical encryption key. Other diagrams, like where do the different keys reside, would also be helpful...to me in particular...as I am constantly getting those wrong when answer questions.
• Add glossary #63 A glossary to the wiki, as some terms are easy to mix up. I would love it if glossary terms were automatically annotated/tool-tips so when a user is reading through, if they are confused on a term they could mouse over and be reminded what it is without loosing there spot. This would require some development with Jekyll plugins and HTML templates for the theme.

[jtmoree-github-com]

I like the idea of linking wiki docs to source code to supplement high level documentation. I have been updating the docs as I figure out how heads works and waiting for feedback before creating PRs. I just posted 2 more.

#67 is for new questions to the FAQ. These are questions I have as a new user to heads. Many are still unanswered. Hopefully we can fill more in during the review process.

#66 is for a bit of re-org on the wiki and new pages for recovery shell and OS installs.

[jtmoree-github-com]

#64 On the front page of the wiki, there should be diagram explaining the ring of trust the exists between the Heads Rom, TPM and the physical encryption key. Other diagrams, like where do the different keys reside, would also be helpful...to me in particular...as I am constantly getting those wrong when answer questions.

I would create such a diagram but I don't understand enough to do so. Is there one I can use for reference anywhere? Or can someone create a crude text version of it in this thread that I can turn into an image?

[tlaurion]

I can go iteratively and answer questions as we go.

0. build system is another discussion.

Runtime:

1. coreboot
   Depending of coreboot version and coreboot support for board, measurements happens in bootblock or in romstage.
   1.1 Under coreboot 4.8.1, its easier to track because it was a patch applied to coreboot itself: https://github.com/osresearch/heads/blob/master/patches/coreboot-4.8.1/0000-measuredboot.patch
   1.2 On 4.9+ the codebase was added into VBOOT codebase (references needed)
   1.3 under coreboot 4.12(?) TPM_MEASURED_BOOT was added, relying not relying on the VBOOT+measured boot codebase, to do the measurements.
   The documentation here should answer questions: https://doc.coreboot.org/security/vboot/measured_boot.html
   Coreboot 4.8.1 patch was modified to push all measurements under PCR2 so that measurement schemes of Heads are the same across VBOOT+measured boot or TPM_MEASURED_BOOT, where Heads uses additional PCRs to measure additional stuff and states: https://osresearch.net/Keys/#tpm-pcrs

2. Heads fires up (which is basically a linux kernel and an initrd. Heads scripts and binaries are included in the initrd)
   A decompression example is here if someone wants to reverse what is built inside of Heads Makefile:
   Quest to reduce firmware size heads#590 (comment)

3. initrd is decompressed, Linux kernel is booted.
   We enter Heads from init:
   https://github.com/osresearch/heads/blob/master/initrd/init

4. Measurements from coreboot, depending on TPM patches/TPM_MEASURED_BOOT are taken.
   (use of tpm internal binary from heads tpmtotp module: https://github.com/osresearch/heads/blob/master/modules/tpmtotp, used as a base for HOTP operations, from https://github.com/osresearch/heads/blob/master/modules/hotp-verification)
   Then:
   4.1 cbfs-init if coreboot as explained above to use what was saved as user file to be measured inside of PCR7(? from memory here)
   https://github.com/osresearch/heads/blob/master/initrd/init#L47-L49
   4.2 combination of /etc/config.user if found in cbfs from cbfs-init
   https://github.com/osresearch/heads/blob/master/initrd/init#L92-L93
   4.2.1 which calls combine_configs https://github.com/osresearch/heads/blob/master/initrd/etc/functions#L266-L268
   4.3 Then booting of board configured bootscript:
   https://github.com/osresearch/heads/blob/master/initrd/init#L57
   Then we split up between generic-init (legacy) and gui-init scenario (or even network-init-recovery):
   4.3.1 https://github.com/osresearch/heads/blob/master/initrd/bin/gui-init
   4.3.2 https://github.com/osresearch/heads/blob/master/initrd/bin/generic-init
   4.3.3 https://github.com/osresearch/heads/blob/master/initrd/bin/network-init-recovery

5. Then if we concentrate only on gui-init
   5.1 for usb boot was explained here: Wiki needs improvements #25 (comment)
   5.2 default boot when /boot detached sign is verified, HOTP remote attestation based on TPM measurements leads to:
   5.2.1 generation of /boot digest, verification of integrity against detached signed version of it
   5.2.1.1 failing codepath
   5.2.1.2 sucessful codepath
   5.2.2 HOTP sucess/fail codepath
   5.2.3 TPM unsealing of Disk unlock key codepath
   5.2.3.1 extraction of LUKS header
   5.2.3.2 unsealing of TPM nv region with firmware measurements + LUKS header + Disk Unlock Key passphrase input from user
   5.2.4 kexec call to OS with/without Disk Unlock Key passed to OS as kernel additional initrd

@ jtmoree-github-com Please ask questions on precedent points.

[jtmoree-github-com]

Started a glossary as I work through some diagrams. All can be reviewed here: https://drive.google.com/drive/folders/14TSM4U0fNYlQpzQEDnpZGCc8eAGDZhSl?usp=sharing

Still early on the diagrams but if you start with HEADS OVERVIEW you will get the idea of where I am going with this.

[tlaurion]

I would have lots of comments and corrections to propose for each diagram.

Can those images be in a shared doc with shared writing access? A pad, or a PR, or a cryptpad.fr?

[jtmoree-github-com]

You should be able to comment now. I can add individual gmail accounts as editors too.

[tlaurion]

I will try to find time in the next coming days and post comments here in regard of filenames on google drive.

[tlaurion]

@jtmoree-github-com A PR with those graphics that will change with Q&A flow would be helpful. Otherwise, I see involvement in all PRs and issues closed will really be time consuming correcting misconceptions... clarified in code.

[tlaurion]

At this point, I belive you should just branch the current master of the PR, and ask question in code. Reviewing the diagrams and current state of https://docs.google.com/document/d/1F7SeHrgYFbbhdBT0fsvddIDvkv0DlnTvjtbTQA00wec/edit#heading=h.5h6usohk6neg as a start doesnt feel like a Frequently Asked Questions at all, but personal questions torward the code, in which I could probably invest 10 hours or more explaining everything. Who is the audience?

[tlaurion]

First review directly on drawings and texts directly

[tlaurion]

I can go iteratively and answer questions as we go.

1. build system is another discussion.

Runtime:

1. coreboot
   Depending of coreboot version and coreboot support for board,  measurements happens in bootblock or in romstage.
   1.1 Under coreboot 4.8.1, its easier to track because it was a patch applied to coreboot itself: https://github.com/osresearch/heads/blob/master/patches/coreboot-4.8.1/0000-measuredboot.patch
   1.2 On 4.9+ the codebase was added into VBOOT codebase (references needed)
   1.3 under coreboot 4.12(?) [TPM_MEASURED_BOOT](https://github.com/coreboot/coreboot/commit/c79e96b4eb310db9d44e36e2dff072c01469c380) was added, relying not relying on the VBOOT+measured boot codebase, to do the measurements.
   The documentation here should answer questions: https://doc.coreboot.org/security/vboot/measured_boot.html
   Coreboot 4.8.1 patch was modified to push all measurements under PCR2 so that measurement schemes of Heads are the same across VBOOT+measured boot or TPM_MEASURED_BOOT, where Heads uses additional PCRs to measure additional stuff and states: https://osresearch.net/Keys/#tpm-pcrs

2. Heads fires up (which is basically a linux kernel and an initrd. Heads scripts and binaries are included in the initrd)
   A decompression example is here if someone wants to reverse what is built inside of Heads Makefile:
   [osresearch/heads#590 (comment)](https://github.com/osresearch/heads/issues/590#issuecomment-623143332)

3. initrd is decompressed, Linux kernel is booted.
   We enter Heads from init:
   https://github.com/osresearch/heads/blob/master/initrd/init

4. Measurements from coreboot, depending on TPM patches/TPM_MEASURED_BOOT are taken.
   (use of tpm internal binary from heads tpmtotp module: https://github.com/osresearch/heads/blob/master/modules/tpmtotp, used as a base for HOTP operations, from https://github.com/osresearch/heads/blob/master/modules/hotp-verification)
   Then:
   4.1  cbfs-init if coreboot as explained above to use what was saved as user file to be measured inside of PCR7(? from memory here)
   https://github.com/osresearch/heads/blob/master/initrd/init#L47-L49
   4.2 combination of /etc/config.user if found in cbfs from cbfs-init
   https://github.com/osresearch/heads/blob/master/initrd/init#L92-L93
   4.3 Then booting of board configured bootscript:
   https://github.com/osresearch/heads/blob/master/initrd/init#L57
   Then we split up between generic-init (legacy) and gui-init scenario (or even network-init-recovery):
   4.3.1 https://github.com/osresearch/heads/blob/master/initrd/bin/gui-init
   4.3.2 https://github.com/osresearch/heads/blob/master/initrd/bin/generic-init
   4.3.3 https://github.com/osresearch/heads/blob/master/initrd/bin/network-init-recovery

5. Then if we concentrate only on gui-init
   5.1 for usb boot was explained here: [#25 (comment)](https://github.com/osresearch/heads-wiki/issues/25#issuecomment-769364795)
   5.2 default boot when /boot detached sign is verified, HOTP remote attestation based on TPM measurements leads to:
   5.2.1 generation of /boot digest, verification of integrity against detached signed version of it
   5.2.1.1 failing codepath
   5.2.1.2 sucessful codepath
   5.2.2 HOTP sucess/fail codepath
   5.2.3 TPM unsealing of Disk unlock key codepath
   5.2.3.1 extraction of LUKS header
   5.2.3.2 unsealing of TPM nv region with firmware measurements + LUKS header + Disk Unlock Key passphrase input from user
   5.2.4 kexec call to OS with/without Disk Unlock Key passed to OS as kernel additional initrd

@ jtmoree-github-com Please ask questions on precedent points.

@jtmoree-github-com want to ask questions on the precedent points prior of going forward so I can modify precedent post and address questions? Edited with TPM_MEASURED_BOOT implementation change directly from coreboot merge.

[jtmoree-github-com]

At this point, I belive you should just branch the current master of the PR, and ask question in code. Reviewing the diagrams and current state of https://docs.google.com/document/d/1F7SeHrgYFbbhdBT0fsvddIDvkv0DlnTvjtbTQA00wec/edit#heading=h.5h6usohk6neg as a start doesnt feel like a Frequently Asked Questions at all, but personal questions torward the code, in which I could probably invest 10 hours or more explaining everything. Who is the audience?

Documenting all in code is difficult because a specific file in source code is only one part of a larger system. The wiki and FAQ address higher level concepts which link to specific implementations in source.

If I have questions after looking at docs AND code then other people might have the same questions. As the questions are answered I am curating the FAQ to small answers with links to wiki and--as discussed--our goal is that the wiki links to github source code.

The diagrams are incomplete as we just started discussing this. It will take some time to flesh those out especially since I am still understanding the workflows and using the FAQ process to do so which takes time as well.

I think I can grant public access to edit the diagrams but that seems like a bad idea given that I am posting the link to this public forum. I'm open to using a different diagramming tool to make collaboration easier. (cryptpad.fr seems lacking in drawing features. perhaps draw.io or lucidcharts?)

[jtmoree-github-com]

I think we need to pause and look at some challenges with the documentation process. Please read this entire post before replying on specifics as the specifics may span multiple areas of discussion--like the docs we are working on. Please forgive me if any of this sounds critical. I am attempting an analysis of the documentation process--not blaming individuals for choices.

As we have all said, the docs need a lot of work. I have 3 pull requests related to documentation that are at least a few weeks old. Ideally, a PR would get reviewed and merged or rejected in a short time frame so everyone can move on to other tasks. This post is to discuss changes in my approach to updating docs that may help remove or mitigate some of these bottlenecks. These changes are in response to what I think are root causes.

I do not have enough understanding of the heads ecosystem to create 'perfect' docs. Others that do, may not have the time or lack tools and/or skills. This effort will take collaboration. This necessitates that we pass 'work in progress' back and forth before it is 'finished'.

Here are the challenges identified so far:

• constrained resources
• scope
• documentation vs executables
• work in progress vs finished work
• tools

constrained resources (free time)

Since most if not all us are working on this in our free time there is a long lag in between work. We all work at different times and days. This encourages parallel work because I am waiting for at least a few days for responses on 'this section' and I only have 'right now' to work on this so I switch to the 'other section'. Of course the other section has questions that I also have to ask and work through responses. This contributes to the confusion. I have seen answers to a question in one PR end up in the other PR because they overlap in concept even though they target different specific changes.

I belive all of this highlights the even greater need to reduce waste and improve communication and collaboration.

scope

The scope of changes I am working on is much larger than an agile practice would call for. This is wasting the limited time that others have by causing them to switch contexts and adding to confusion around the questions I am asking. The scope for 2 of 3 PRs is too large.

In some cases the questions and answers need to stay very high level. While I appreciate the very detailed responses, there are times when a short sentence or confirmation is all that is needed. In addition, when the long answer is repeating content it frustrates everyone--especially the poster who feels like his time is being wasted.

This is a significant challenge for documentation as it--by design--references multiple areas within the project in order to relate them to each other. In some cases the folders and pages need to be re-organized. e.g. allow room for growth, etc.

We will need to work on keeping scope small for a specific PR to keep it manageable.

documentation vs code

Since this project is based around executable code and we are using tools built to manage code we are also treating documentation as code. In a literal sense this may be true but the workflow and approach to documentation should probably be different. Concepts that are new to users may take only a paragraph to explain usage but might take many different components, processes, utilities, and user interfaces to implement.

My focus has been on the 'new' experienced user since that is my use case. Having worked in I.T. for a few decades I have a broad understanding of the tech involved but also need to study and troubleshoot specific concepts and technical issues which requires finding information in a targeted way.

I have been finding this information by reading websites, code, docs, and posts to github. This has been a slow process which illustrates the points I am making here. There have been a few times that I asked a question that was already explained clearly in the heads wiki or docs but most of the questions are hard to understand without more information. I created a list of questions when I started digging into this project and as I find answers to them I am integrating those answers into the Wiki. This adds to the scope creep mentioned above.

1. THIS IS NOT HOW CODE WORKS

Code either works or doesn't--there is not much room for progressive elaboration. At most, we could compare this to changing code in response to feature requests but really, these are just different.

2. Embedded Docs

I like adding documentation to code--in fact, any source code I have created in the last 5 years includes more lines of comments than executables (generally as a header at the top). I will continue to attempt to add documentation to heads utilities but there has been pushback on that for valid reasons. Any documentation has to be updated but the only alternative is No documentation which is not a good alternative.

3. Design Decisions

I often search other websites for the answers to my heads questions. Reading code doesn't tell me definitions for acronyms and other higher level stuff such as why does heads use a TPM for a disk key instead of just encrypting it and storing the file in /boot? These types of questions are going to come up for people who are getting familiar with the project. Especially, as multiple ways of handling a use case are added. e.g. verification with HOTP vs TOTP, disk key in TPM or not, etc.

work in progress vs finished work

Generally, a PR is the end to a finished (or almost) finished process. In the case of 2 of my 3 PRs we are working through the final issues that are blocking them from being merged. I think these two are also getting blocked by the 3rd (FAQ) because they deal with the same issues of documentation.

1. more information needed

Given the nature of a normal PR, when I post questions to reviewers the responses seem to assume that the work is finished. In almost every case I need more information and I am asking for clarfication. I wait to commit the changes until the questions are answered but the responses seem agitated that I have not already 'fixed' the problem with another commit. This sometimes encourages the replies to repeat information because it seems like I have not read or understood the first post. This is especially bad when there is copy/paste code where two utilities work almost the same way and both need modification in the same way. I generally, will not post on the second until the first is resolved even though they are in the same PR.

2. process and labels

The FAQ PR is a draft because it is on ongoing collaboration. I created it at the suggestion of others on this thread. There are other ways to collaborate though they also have challenges. We may find it helpful to change our approach to Work In Progress. I am open to suggestions.

We have to be able to post W.I.P. and get feedback on that work. Ideally, multiple parties edit the work. Maybe that is where the PR breaks down. Instead of editing the work, reviewers are posting metadata about the work which then has to be interpreted by me and incorporated and reviewed again. If we all edit with revision history it might go faster.

tools

1. github

As a github project GITHUB is our main collaboration tool. I am happy to use any tools the group feels are necessary and helpful. As I already mentioned, a Pull Request may not be the best tool for work in progress. Unfortunately, I don't see any other tools working better. Perhaps by marking the PR as a draft it will help.

2. alternatives

We could try to move the documentation to another system but that seems like a really big hurdle to cross and I'm wary of adding dependencies by moving the docs outside of the current systems.

3. diagrams

The diagramming work is new and requires new tooling in addition to all the same collaboration problems: it is NOT finished; it is highly technical but needs to start with broad concepts; it requires write access for multiple parties to be most effective. etc

Again, open to suggestions.

Action Items

I resolve to follow these guidelines as I continue to work on heads. Though it will slow down the potential pace of development the development is slower anyway due to confusion. I may be able to revise these after we work as a team and understand each other better.

• I will reduce scope on changes if possible
• I will not submit multiple posts and changes in parallel
• I will not publicly post work in progress
• I will clearly label comments as WIP or NEEDS CLARIFICATION on a PR

[jtmoree-github-com]

@jtmoree-github-com: I'm sorry, but at least half of this post was completely unnecessary. This is mostly just empty speech for me.

You could at least comment on the half that was necessary... Perhaps just concentrate on the Action Items. Do they seem useful?

[tlaurion]

@GittCatt I would love to know what was irrelevant to you and what your audience is. Please select the irrelevant parts in code (selecting the lines as a block) at least, otherwise those kind of comments are discouraging people actually contributing, who are trying to make it better by answering their own questions.

[tlaurion]

@jtmoree-github-com maybe a riseup pad would help getting input on content only? I can commit reviewing written stuff wherever it is hosted at the beginning of the week.

My intuition is that PR might not be the best way to do this. But as said previously, I suck into bringing technical to non technical. My job here is to make sure what is published is true and technically sound.

[tlaurion]

@jtmoree-github-com maybe a riseup pad would help getting input on content only? I can commit reviewing written stuff wherever it is hosted at the beginning of the week.

My intuition is that PR might not be the best way to do this. But as said previously, I suck into bringing technical to non technical. My job here is to make sure what is published is true and technically sound.

[qubicrm]

@jtmoree-github-com Right now, it seems to me that the most difficult thing to grasp from the documentation is the fundamental differences between "standard" and maximized boards. I have been reading through the issues, but still have unresolved questions (probably because I am doing something wrong, after I flash skulls, then flash X230, everything OK, but when I flash x230-maximized (through heads graphical UI) a brick results). At first I thought it was related to MAC address, but does not seem that way. I checked that I unlocked IFD in skulls script: yes. tried flash x230-maximized from circleCI: no reward. tried to recreate gbe.bin with the machine MAC: brick. And still wandering if I am doing some silly mistake
Now, going to try to flash the maximized roms directly (not through skulls).
If you some error in my reasoning please ping. But I just wanted to give you guys feedback.

[jtmoree-github-com]

@qubicrm I have not had to deal with that issue as I use a Purism laptop that already works with heads. My focus has been on the installation of an Operating System and Troubleshooting boot. These are tasks that are performed after a working heads installation. I will post an update on the work I am doing which may help if you learn some things and decide to share that information and/or improve the wiki.

As I mentioned in my action items above I am solely working on 1 pull request right now (docs refresh). The key point that everyone should know is that Pull Request workflow is not conducive to collaborating on documentation. In the future I will be using other tools like the ones mentioned in this post. That isn't new information for this thread but is important.

An analysis

A wiki is designed to be collaboratively edited inside a web site but that is best scoped at the context of a 'page'. As changes span across multiple 'pages' the wiki approach becomes less effective. Github allows all of the wiki 'code' to be edited outside of the wiki in a git repo. While this makes it easier to edit more than the 'page' context, it complicates collaboration.

When we need to collaborate on the content going into multiple pages on the wiki we are encouraged by the wiki docs themselves and the features in github to fork. We then get a personal copy and can configure github to show our copy to others. This is great for review but does not allow for easy collaboration on edits.

In my case I reorganized large sections of the wiki and am still learning and understanding the heads environment. This led to many rounds of commits, change requests, rewrites, and repeats. At this point we have 9 files changed and 20+ commits in my PR. In a typical PR situation this would be rejected as too complex. In fact, I may be breaking the PR into multiple smaller PRs for the final merge.

It is the wiki that is collaborative--not the individual edits. While it is easy to say, "Make smaller changes" that is challenging when the smaller changes link to each other and move content around. I had to have a working wiki with all of the content arranged in order to do the work of updating. The other challenge is one of scope. Different opinions exist in the project about what is in or out of scope for the wiki.

What will I do for the next changes I work on which span multiple pages that might improve collaboration?

1. clearly identify the intentions of my change and clarify scope in an issue
2. after the fork I will make edits and check in so that I have a 'copy' of the wiki in my own github account
3. If I suspect that others will suggest changes I will create a collaborative space where we call all edit content at the same time
4. I will post to the issue asking for review rather than a PR and suggest that people copy the content to the other space for collaboration

This approach has the disadvantage of losing all of the organization and diff tools that github includes and we may find that this has a different set of challenges. Naturally, the smaller we can keep changes the less problems we will encounter but the point I am making is that not all changes are small.

[tlaurion]

@qubicrm sorry about this confusion.

As written in the notes of the maximized PR, coming from Skulls requires the initial additional step of unlocking IFD from their documentation and then a full flash, directly with flashrom from Heads recovery shell (mount-usb; flashrom --force --noverify-all -p internal -w x230-maximized-12mb.rom) , not from the helper scripts from the flash menus, to write the whole 12MB flash rom. Only then, when within maximized Heads ROMs, can the helpers scripts from Heads menu be used to upgrade Heads internally.

Otherwise, it requires an initial external flash, since ME and IFD are locked and only accessible in read only mode from the booted system.

Maximized builds are basically to be flashed externally, once, by flashing the 4MB rom on the top SPI, and the 8MB rom on the bottom SPI flash chip, which contains neutered ME and unlocked IFD, which replaces flashrom options inside of the board config to not only flash the BIOS section of coreboot internally on future internal flash attempts.

TL;DR: You are causing a brick because the IFD from Skulls is defining a smaller BIOS region then the one being flashed by the maximized rom, causing inconsistencies between regions defined under IFD and beginning and end regions really being on opaque combined 12MB SPI flash chips seen internally. Basically, flashing the bigger BIOS region from maximized builds invalidates expected structure from IFD, which is lost and fails.

The solution here would be to flash both SPI chips with related ROM images (top and bottom). Externally.

I am more then due to document this properly, and will try to do so ASAP to fix #55

[tlaurion]

@jtmoree-github-com this PR is near completion IMHO in terms of content.