Add xx30-maximized and xx30-hotp-maximized boards (11.5mb flashable BIOS regions, reproducible me.bin and generated gbe.bin and totally externally and internally flashable roms)

[tlaurion]

REQUIRES:

• External flashing with individual maximized-top.rom on top SPI chip and maximized-bottom.rom counterpart on bottom SPI flash chip.
  or
• Internal flashing over initially unlocked ifd (did you run ifdtool -u on your bottom.rom stock backup prior of reflashing it?).
  If so, internal flashing is possible through a manual flashrom --force --noverify-all -p internal -w maximized.rom (12mb rom) from command line to switch from xx30 to xx30-maximized.

Adds:

• blobs/xx30/me.bin (98k, ROMP and BUP only modules, cannot be removed), shared among all xx30 boards (Helix (Type 3xxx), T430, T430i, T430s, T430si, T431s, T530, T530i, W530, X1 Carbon (Type 34xx), X1 Helix (Type 3xxx), X1 Helix (Type 3xxx) 3G, X230, X230i, X230 Tablet, X230i Tablet, X230s
• blobs/xx30/ifd.bin (Might be generalizeable to ifd.bin: haven't tested on above boards EDIT: Reported working)
• adds coreboot config which specifies maximal CBFS space usable inside of Heads (11.5Mb instead of 7Mb)
• GitlabCI (EDIT: changed their caching system. Unusable for our use case now) and CirecleCI modifications, changing the default board compiled to be x230-external-flash (instead of x230) EDIT: no replace.
• CI downloads build dependencies, enforces cache (GitlabCI, CircleCI (1)(2)) and quick fixes (cleans make build and board local builds that are not properly declared in Makefile... To fix later, just want it to work right now)

CI:

• Add build logs as artifact to more easily compare non reproducible builds.
• Add initrd.cpio.xz to facilitate build reproducibility troubleshooting between CIs builds.

Permits:

• deprecation of x230-flash board currently required for initial internal flashing of x230.
• Internal flashing of the whole 12Mb flash image and not only the BIOS region. Requires fully valid ROM (no faked IFD as currently used for xx30 boards) for subsequent ROM updates, since board flashrom option does not specify BIOS region anymore for subsequent flashrom upgrades.
• Maximal space for other tools integration.

Tested on x230 and x230t from latest commit CI build's artifacts. (Click on green checkmark to go on CircleCI, go in the artifacts section and download ROMs (both top and bootom roms to externally flash, or, full 12mb rom to flash internally, see REQUIREMENTS)

Tagging all x230 owners as per #691 : @osresearch @merge @jan23 @flawedworld @MrChromebox @shamen123 @eganonoa
T430 owners: want to adapt and test if x230-ifd.bin can be used on that board too? EDIT: reported to work.

Would close #307.

Edit: put links everywhere to facilitate understanding implied changes.

Edit: As of now, this concept as been generalized to xx20 and xx30, with boards definitions having been created for x230-maximized, x230-hotp-maximized, t430-maximized, t430-htop-maximized, t420-maximized, t420-hotp-maximized, x220-maximized, x220-hotp-maximized.

Where:

• -maximized boards: Includes xxx0 ivy/sandy bridge initial board configs with additions which required additional space that were deleted over time under xxx0 initial board counterparts. For exemple, the initial board configs won't integrate cryptsetup2, might revert to gpgv1 if needed in later time. Remote attestation happens solely over TOTP at each boot.
• -hotp-maximized boards: includes the above, with an additional dependency in bootpath against HOTP remote attestation, which depends on users having a HOTP compatible USB Security dongle (Librem Key, Nitrokey Pro, Nitrokey Storage) to challenge TPMTOTP measurements over HOTP with the USB Security dongle. As a result, both Heads and the USB Security Dongle will report the result of the remote attestation challenge. On screen with HOTP:Success while the LED will flash green. It is also to be noted that as opposed to boards not having a TPM, that challenge will fail if going into recovery and going back to gui-init, since the PCR being linked to Heads state (normal, recovey) will change and will invalidate measurements challenged at remote attestation.

[tlaurion]

I could rename x230-external-flash board to x230-neutered_me if your prefer. EDIT: renamed maximized
@osresearch : Following #307 discussions, it is impossible to download ifd, but possible to download ME and produce me.bin from me_cleaner.

If you have any advice into putting innoextract calls into a module Makefile, I could remove ME from the xx30 directory and only have x230-ifd.bin (that might generalize into ifd.bin after other xx30 boards are tested by community.)

Waiting on your input guys!

[tlaurion]

@osresearch @merge @jan23 @flawedworld @MrChromebox @shamen123 @eganonoa
Please test and comment.

[tlaurion]

Will merge myself in the next coming days. This project stands still and this is really, really sad.

[MrChromebox]

I'm not able to review the CI bits, but as for the rest:

• blobs look sane
• why a separate linux config for this board? it's exactly the same as linux-x230.config -- why not just use that?
• I realize you just duped/edited the coreboot-x230 config, but there's a lot in there that doesn't make sense / isn't needed:
  -- why not include the config file in CBFS?
  -- why not include timestamps in cbmem?
  -- USE_BLOBS is useless since it only affects coreboot checking out the blobs repo, which we do manually
  -- deselecting POST_IO and POST_DEVICE is better done via CONFIG_NO_POST=y
  -- makes no sense to select DRIVERS_UART_8250IO then disable CONSOLE_SERIAL
  -- DRIVERS_PS2_KEYBOARD isn't needed for linux payload
  -- UART_PCI_ADDR is default and doesn't need to be set
  -- PXE is deselected by default, no need to explicitly deselect
  -- no need for DEBUG_SMM_RELOCATION
  the above changes drop the config from 29 lines to 19
• for the ROM split, any particular reason to use a 64kb block size, vs say 4M (assuming you wanted to use the same for both)?

[tlaurion]

@MrChromebox : thanks for the review!

why a separate linux config for this board? it's exactly the same as linux-x230.config -- why not just use that?

Agreed. More modules inclusion in the main linux config to come. Actually, I would go forward and also upgrade the kernel to at least the latest LTS kernel of that branch, or even better, move kernel support to 5.x branch. Will do in seperate PR. The reasoning on this is that the x230 board has actually changed in time. For example, the MMC controller passed from Ricoh to Realtek. Unfortunately, Realtek MMC kernel support is not in the current kernel nor config.

In the meantime I will point to the linux-x230.config

I realize you just duped/edited the coreboot-x230 config, but there's a lot in there that doesn't make sense / isn't needed:
-- why not include the config file in CBFS?

Added

-- why not include timestamps in cbmem?

Added

-- USE_BLOBS is useless since it only affects coreboot checking out the blobs repo, which we do manually

Added

-- deselecting POST_IO and POST_DEVICE is better done via CONFIG_NO_POST=y

Removed

-- makes no sense to select DRIVERS_UART_8250IO then disable CONSOLE_SERIAL

Removed

-- DRIVERS_PS2_KEYBOARD isn't needed for linux payload

Removed

-- UART_PCI_ADDR is default and doesn't need to be set

Removed

-- PXE is deselected by default, no need to explicitly deselect

Removed

-- no need for DEBUG_SMM_RELOCATION

Removed

the above changes drop the config from 29 lines to 19

Same result

for the ROM split, any particular reason to use a 64kb block size, vs say 4M (assuming you wanted to use the same for both)?

Just reused an old recipe of Trammel that just worked (in tree) that didn't made any sense back then because of faked IFD and absence of ME, creating invalid 4Mb and 8Mb images from invalid 12Mb image (since back then an external flash map was used to instruct older version of flashrom to only flash BIOS region of it, which is now replaced by flashrom IFD statement specifying BIOS region. With blobs included, we remove the need to specify that IFD BIOS region since the image is valid and can be flashed internally as a whole. It BS size doesn't really change anything really here, unless you have a concise reason to change it. The important part is the skip statement, the rest is just arithmetic.

Edit: those statements produce the actual externally flashable rom images, removing the need to have the x230-flash board config and produce those top and bottom rom images from coreboot.rom which is flashable internally, using flashrom differencial against 12Mb opaque combined SPI flash. The goal of this board proposition is exactly that: having a 12Mb image which can be verified/flashed internally while removing first flash requirement with x230-flash board config. Did I missed your point?

[tlaurion]

@MrChromebox

• Tested CI build freshly produced by artifacts attached to this PR on x230t flashing both bottom.rom and top.rom: Works.
• Tested flash.sh to flash whole 12mb coreboot.rom internally: Works.

[MrChromebox]

assume these commits will get squashed when merged, so doesn't really matter than the config delete is here vs the other commit

[tlaurion]

Received FSFE Legal advice today. Read here

[tlaurion]

@MrChromebox : following the legal advice above. Is there any trick i've missed that would permit to use innoextract or cabextract against the ME executable to automatize this without having to host the blob? @osresearch was ok fdor github to distribute IFD but not so confortable distributing ME.

I have no clue how to accomplis this.

[MrChromebox]

@tlaurion you had ME extraction working from FL2 via innoextract per your comments in #307 here, is that not sufficient?

[tlaurion]

Process is documented in readme to reproduce me.bin, not automatized. As a result the me.bin is in the PR, not code that would produce that me.bin without hosting it.

On April 19, 2020 6:06:14 PM UTC, MrChromebox ***@***.***> wrote: @tlaurion you had ME extraction working from FL2 via innoextract per your comments in #307 [here](#307 (comment)), is that not sufficient? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #703 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[tlaurion]

@tlaurion you had ME extraction working from FL2 via innoextract per your comments in #307 here, is that not sufficient?

@MrChromebox: still trying to find a way. New attempt is to make a xx30 submodule depending on me_cleaner and have innoextract downloaded on host system (CIs). Trying to make it work.

[tlaurion]

Seems like without gbe blob, no ethernet.

@hughies? Could we extract that from LVFS archive from a download script without hosting the blobs ourselves anywhere?

[MrChromebox]

since this is going in after the flashrom 1.2 bump, can update CONFIG_FLASHROM_OPTIONS to drop the unnecessary params now

[tlaurion]

@MrChromebox : Will change FLASHROM_OPTIONS, thanks for the heads up.

But with gbe now thought to be needed to be inserted in coreboot to have a working ethernet port on final host system, there is still be work needed, added with live extraction of ME and neutering+deactivation of while unlocking of ifd from CIs for this PR to be pertinent...

There is still work needed to go forward with this PR:
1- Can we distributed Gbe ...
2- Can it be extracted from online source (not horsted on Github) from CI, and be reinjected in ROM so that #700 is not a problem...
3- Can we fake the mac and inject it or wipe the mac from CI, wipe gbe while still having a functional ehternet discovered by e1000e

[MrChromebox]

it seems like we're jumping thru a lot of hoops to try and build a reproducible full flash image, rather than a reproducible coreboot/Heads one. Any reason we don't simply focus on the BIOS region only, and let the IFD/ME/GBE be the user's (or a downstream) problem?

[tlaurion]

@MrChromebox The problem i'm attacking here is to be able to add more stuff in the BIOS region, which is actually limited to 7Mb and near saturation, while permitting CIs to prove that SPI content and bult reproducible images ARE reproducible and that the content on SPI flash IS identical and verifiable on unlocked IFDs to permit reflashing/read/write of whole 12Mb opaque flash chip.

• use coreboot vboot measured boot #562 and consequently Enable vboot on Lenovo devices for measured boot #709 (see vboot-*-me boards) cannot have VBOOT added as a result as of now, which leads to Quest to reduce firmware size #590 which will at best following readings I have done, without LTO in but whith "hammering" option, could lead to an expected 11% of additional space currently occupied by the kernel.
• If we dream a bit, and open the door to CPython integration help needed #689 or LVM thin-provisioned volumes #202 support, or want to open the door to remotely/locally verify LVM or any other task linked to integrity/security audit (Import chipsec tool and make it work in the recovery shell #231) this won't be possible, which is why u-root and other projects are becoming interesting as alternatives to Heads because of code being ran at run time (text files highly compressible) while Heads rely mainly on cryptsetup (u-root inclusion: not interested), gpg(u-root: not interested and point to other projects which would then depend on same native libraries for usb token support anyway).

My goal here is, once again, to give a chance to Heads to do (and continue to do) what it actually does already well and stop relying on patches (measured boot patch -> coreboot vboot+measured boot upstreamed) to do its job, while maximizing available space for other tool integration, while at the same time permitting CIs to build those reproducible images it is supposed to produce (not the case currently #640 #571) reproducibly out of the box. This is multiple birds one stone proposition to permit Heads to attract more developers, both from the community but also to be integrated in other projects and be customized further more, which is right now locked down because no more space is available. As of now, there is no possibility of building x220 nor other boards actually, automatically from CIs. This is problematic on a lot of levels. If we only pick one, no CI, and no LVFS, and no FWUPD for firmware upgrades under QubesOS, to be released soon. We would all benefit to be able to tackle this problem for transparency and not target only high technical people who are able to compile software, verify reproducibility (where?) and manually upgrade their firmware internally.

Continuing with using only the BIOS region (7Mb CBFS region not touching ME) path is not only limiting, but doesn't permit to neither internally validate signed OEM rom image completely, nor verify ME region against external source, nor would permit FWUPD to completely reflash/upgrade the whole 12mb flash, but would require the user to be tech-savvy/blind trust us as OEMs, which is not really consistent with Heads promises and leaves it to a PoC state or hack our ways to download a whole image from somewhere else and extract the binaries in a script by CIs to accomplish the same result, requiring OEMs reselling other models to miss that transparency point.

The truth is that more CBFS space is needed. Proof here, now.. xx30 and xx20(even more) are stuck at coreboot 4.8.1 if we don't do something about it.

[tlaurion]

For some reason, cache has expired so builds will take longer then expected under last commit linked CI, happening under yellow dot now on last commit.

When done, ROMs, hashes files, build logs and cpioz will be be found under the artifact section on CircleCI

[bwachter]

Is it possible to change the MAC before flashing with a tool like the old ich9gen, or is a local build required for that?

[Thrilleratplay]

@tlaurion While we have talked about the path to migrate Lenovo models xx20 and xx30 to larger Heads BIOS partitions, there seems to be a confusion on what is going on. Given the many moving parts, maybe create a Github project to track the progression of issues/PRs/Questions?

@bwachter As of coreboot 4.13, a bincfg similar to ich9gen has been added to allow generation of the GBE for xx20 and xx30 boards with the ability to change the MAC address. This PR includes a patch which back ports it to coreboot 4.8. Although, this raises the concern that not everyone may want to use the default 00:DE:AD:C0:FF:EE. As it stands right now, the would need either need to build locally with the desired MAC created in the blobs directory or flash the user created GBE partition separately after Heads has been updated. Perhaps this can be simplified for the user by the flash.sh generating a patch of the MAC and checksum from the current GBE to be applied to the written ROM.

[tlaurion]

As of right now @bwachter : you can change the mac here.
And rerun the detailed instructions found here, specifically the instructions to manually generate gbe.bin or the instructions to extract from backup your gbe.bin

[tlaurion]

@tlaurion While we have talked about the path to migrate Lenovo models xx20 and xx30 to larger Heads BIOS partitions, there seems to be a confusion on what is going on. Given the many moving parts, maybe create a Github project to track the progression of issues/PRs/Questions?

Those were specified here, mainly:

Basically, if ifdtool -u was called on backup initially prior of reflashing the 8mb bottom SPI chip, you could adventurously (IF YOU HAVE EXTERNAL FLASHER to flash the two ROMs individually to fix brick) flash internally the whole ROM with:
flashrom --force --noverify-all -p internal -w heads-x230-hotp-maximized-rom_tag-git_commit.rom that will be produced on CircleCI. Note that you have to choose between hotp-maximized or maximized versions, depending if you own a Nitrokey Pro/Nitrokey Storage/Librem Key that you intend to use for remote attestation (aside of its smartcard features).

As for deprecating xx20 and xx30 boards, @flammit showed opposition.
There is also opposition in unlocking ifd, considering ME is not measured right now. (Even though there is no more useful modules to inject whatever from, since only BUP and ROMP are present... Resulting in a region of 9kb of undeletable blobs)

Considering BIOS region is measured, sealed, and revalidated just before boot with Disk Unlock Key passphrase (so nothing went to recovery sed'ing our policies), i would go forward and even implement a flashrom verification of ME partition prior of booting into another PR (with proper additional flag in board config). Otherwise, no other solution were proposed, and xx20 and xx30 platforms, with their limited SPI space being already saturated, I will give the maintainsership of those original platforms (with their x230-flash, x230-hotp and others, without neutered ME and that space made available for BIOS region) to someone else.

I'm out of ideas, like said so many times before under #590 and all tagged issues.

[tlaurion]

Al always, I'm open to debate and go forward with constructive solutions.

[tlaurion]

x230: @tlaurion @osresearch @merge @jan23 @flawedworld @MrChromebox @shamen123 @eganonoa @bwachter @Thrilleratplay @SebastianMcMillan
t430: @Siproqu

Testers: Ready to merge unless told otherwise.
You can check building recipe here used for CircleCI:

• https://github.com/osresearch/heads/pull/703/files#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R11 and
• https://github.com/osresearch/heads/pull/703/files#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R242-R362

Look at the build results here
And download the CircleCI roms produced here

Planning to merge this at the same time as #912 :
Ready to merge unless testers say otherwise.
t420: @SebastianMcMillan alexmaloteaux
x220: @SebastianMcMillan @techge @eganonoa @shamen123 @Thrilleratplay @userongihu @BlackMaria

[fhvyhjriur]

It is impossible to unlock the IFD/ME, even with 1vyrain or other things when i only flashed the 4M top-chip with coreboot/skulls? In https://github.com/merge/skulls there is written that flashing the 8M-Chip is optional and that is why i have not flashed it. I also gave my external flasher to other people and dont have it any more because i thought i wont need it any more. Now i am not able to use the heads xx30-maximized images?

[MrChromebox]

@fhvyhjriur that's correct. The IFD can be unlocked only via external flashing, and doing so is required to use the -maximized Heads images

[tlaurion]

Ok. This is ready to merge to my eyes!

@MrChromebox ? Some things my eyes missed?

The magic operates here, prior of each maximized board being compiled.
If the user wants to use his own me.bin, gbe.bin and ifd.bin, he can call extract.sh instead or follow README manual recipe.

[flammit]

Except for the NITs and extra file - this LGTM (without having tested it on real hardware).

Please also squash the commits before merging there is way too much noise for what is a straight forward blob/boards addition.

[tlaurion]

Misses documentation linuxboot/heads-wiki#55

[tlaurion]

See linuxboot/heads-wiki#25 (comment)

Make sure to address @qubicrm linuxboot/heads-wiki#25 (comment)