libremkey-hotp-verification is not reproducible

[tlaurion]

Side details:

• libremkey is a checkout of nitrokey, renaming directory. Name should be vendor neutral if there is agreement.
• vocabulary used under Heads should be consensual on what should be used to refer to actually used technology and not confuse users furthermore. Is it "HOTP-validating USB security dongle" as opposed to "OpenPGP security dongle" when HOTP visually attesting security features is to be used under Heads?

Technical details here:

• EDIT: The trick applied upstream to fix commit IDs (download specific commit ID related github archives) instead of cloning from git, cannot be applied because it relies on hidapi submodules. That needs to be fixed since, once again, upstreamed version changed last week and CI caches builds unless cache are flushed manually and restarted, which can then builds from scratch, else, git is not redownloading, which is problematic.

On local copy of Heads which was already used to build, that means:
make BOARD=YourBoard real.clean && rm -rf crossgcc && rm -rf build/*/.

Else stating that your builds are working on latest Heads commit id is meaningless and won't produce the same hashes for the same board.

Tasks:

• Agree on vocabulary to be used to talk about GPG card. (OpenGPG security dongle, USB security dongle, USB security key, Heads security dongle/key. Unify language to be used also per manufacturer to describe the product name.
• Find a way to pin libremkey-hotp-verification to a specific commit ID, including it's dependency to hidapi

[tlaurion]

@kylerankin please reply once clarified with Nitrokey.

[kylerankin]

@tlaurion Could you elaborate on what needs to be clarified with Nitrokey? My understanding was that you and Nitrokey were working out what verbiage you wanted to use. I don't have a strong opinion as long as the term isn't too technical (the initial driver for the Librem Key naming originally was avoiding confusion back when some Nitrokey Pros didn't support this feature and we were the only company supporting this use case with Heads--both are no longer the case), and as long as it is applied consistently throughout the UI.

If the phrase used to describe the category of devices ends up being too technical we may end up sticking with "Librem Key" in the fork we ship with our laptops, since we ship them with Librem Keys and that's the easiest way to describe them to our customers who might have a wide range of technical experience. There is already some confusion out there about Librem Key/Nitrokey vs. a regular USB thumb drive since they have a similar appearance (and since in our case we ship both Librem Keys and USB thumb drives with these orders).

[jans23]

How about a less-technical term which instead refers more to the actual use case:

1. "Heads compatible" - however I doubt that Purism's, Insurgo's and soon Nitrokey laptop customers necessary know what Heads is.
2. Better may be "measured boot supporting USB key" or "measured boot compatible USB dongle".

[elsehow]

I'll preface this by saying that I'm most familiar with the Insurgo reownership use-case, so these notes come primarily from this perspective.

From the user's point of view, the device in their hands should always be called the same thing. Qualifying the term "USB security dongle" with terms like "measured boot supporting" or "boot compatible" may be more complex than required: ultimately, the user needs to know what device to grab, and have enough information to discover what role that device plays in the context of reownership and boot.

In that spirit, the most succinct solution for the Insurgo use-case may be to refer to the device as a "USB security dongle" or "USB boot dongle," and to specify HTOP or GPG functionality in context throughout reownership.

[kylerankin]

and to specify HTOP or GPG functionality in context throughout reownership.

I believe the second part of this is in place throughout most of the code base. I know that in the sections we've contributed, whenever the device is being used for GPG we try to use a generic term like "USB security token" because you could use a Yubikey or any number of other USB security tokens that work with GPG. So far we have said "Librem Key" when referring to HOTP operations, like "Please insert your Librem Key" for the reasons I stated above.

So given that multiple companies are supporting this and that devices that are labeled "Nitrokey" and "Librem Key" on the outside would work, it makes sense to change it, since "Please insert your Librem Key" wouldn't make sense to someone who has a Nitrokey, and "Please insert your Nitrokey" wouldn't make sense to someone with a Librem Key. The only question is whether whatever term everyone agrees on would also apply to a Yubikey ("USB security token") and if that's the case, whether anyone is concerned about possible confusion there when using the HOTP function.

There are also functions within Heads like generating a GPG key or flashing a new ROM that prompt the user to insert a USB disk (usually referred to in the UI as "USB drive") so that's another area where picking the wrong term could cause confusion.

[elsehow]

There are also functions within Heads like generating a GPG key or flashing a new ROM that prompt the user to insert a USB disk (usually referred to in the UI as "USB drive") so that's another area where picking the wrong term could cause confusion.

To my eye, "USB drive" or perhaps "USB thumb drive" are sufficiently common parlance as to not be confused with a "USB security token."

It seems the first order of business is to ask: does anyone disagree with the term "USB security token" to describe both Librem Key and Nitrokey? (We can discuss the HOTP/GPG function once there's consensus).

[jans23]

I don't think the term "token" is good because I experienced people misinterpreting it for a software token, like an access token for OpenID or such. Instead I prefer "key" or "dongle" so that would make it "USB security dongle".

[elsehow]

@jans23 That makes sense to me. I would add that "key" might cause similar confusion (key could be a file, like a PK).

In that case, any objections to "USB security dongle"?

[jans23]

@tlaurion pointed out the risk of confusing it with a Yubikey #641 (comment)

To avoid such confusion we should list the supported devices clearly in the documentation. Any suggestions where exactly?

We could underline this aspect by using the term "compatible USB security dongle" which implicates that not all USB security devices would work.

[elsehow]

I think the README is as reasonable a place as any (and is relatively easy to update).

I think it's reasonable to use the word "compatible" on first introduction, along with a reference of where to find compatible keys for your version. After first mention, the term "USB security dongle" alone will probably suffice.

[jans23]

Sounds good to me.

[alex-nitrokey]

I created a quick variant. Please have a look what you think about it.

This change would allow vendor specific versions of the tool. Maybe it is better to overcome this kind of separation upstream altogether though. I am not sure.

There are atches of the upstream version. I am wondering if we should just include them upstream as well (at least those that makes sense there), instead of patching in heads...

[szszszsz]

Just stumbled on this conversation. Let me reuse my comment from Nitrokey/nitrokey-hotp-verification#11 (comment):

Just an idea by the way: brand can be taken from the used device, and stored for the later use for UI needs. No need to have it hard-coded in multiple binaries, unless there is a need to accept only one particular device model.

That is assuming there is any storage available at the boot time. Additionally, all supported devices could be mentioned during the prompt, since they are listed in the binary anyway:

https://github.com/Nitrokey/nitrokey-hotp-verification/blob/1ae3c5be091a3bd8d2bf2b2469d9694be2be06e2/device.c#L41-L46

[kylerankin]

I like @szszszsz 's idea of taking advantage of the fact that we know the list of supported devices in advance, and each now do have unique USB IDs, so we should be able to detect and present the appropriate device name to the user once it has been enrolled (although for the initial "Insert your ____" warning that might require us to store that state somewhere, either as a config value in the cbfs, which would mean reflashing each time we update HOTP, or possibly in the file that contains the HOTP incrementing counter in /boot).

[mfc]

sorry to jump into this thread late -- I like the simple, generic approach outlined by @elsehow.

why not also include the word "Heads" in the name, given we are discussing a HOTP-enabled USB security key/dongle for use with Heads?

the term is very accurate, vendor-neutral, and most consumers (besides Purism customers) will be aware that Heads is the technology being used to provide boot attestation for them. for Purism, it shouldn't be difficult to inform customers about the technology being used given they are informed about coreboot.

re: dongle vs key, my preference would be Key as people are familiar now with keys potentially being USB sticks (Yubikey, Google Security Key, etc).

[tlaurion]

sorry to jump into this thread late -- I like the simple, generic approach outlined by @elsehow.

why not also include the word "Heads" in the name, given we are discussing a HOTP-enabled USB security key/dongle for use with Heads?

That would bind the technology to Heads only. I have no problem with that personally, but have no idea of the impact of creating a term that would then be spread while the key/dongle could be used in other Open Source Firmware projects outside of Heads. I think personally that the coined term should be more generic, not bound to a software project. (Heads might be refactored and integrated under u-root, for example and might change name consequently.)

the term is very accurate, vendor-neutral, and most consumers (besides Purism customers) will be aware that Heads is the technology being used to provide boot attestation for them. for Purism, it shouldn't be difficult to inform customers about the technology being used given they are informed about coreboot.

re: dongle vs key, my preference would be Key as people are familiar now with keys potentially being USB sticks (Yubikey, Google Security Key, etc).

@mfc, I do not know if you read this reply in this thread. Is your preference the same after reading this comment?

Debate in code

[mfc]

That would bind the technology to Heads only. I have no problem with that personally, but have no idea of the impact of creating a term that would then be spread while the key/dongle could be used in other Open Source Firmware projects outside of Heads. I think personally that the coined term should be more generic, not bound to a software project. (Heads might be refactored and integrated under u-root, for example and might change name consequently.)

yeah i think maybe it's a difference of framing. i see the scope of this issue/effort as trying to create the same terminology for the projects that are currently implementing Heads. something that is so general to be used in other contexts may actual end up being named differently in those other contexts for users to better understand their purpose. i'm not opposed to "USB security dongle", it is still better than having different terms within three projects implementing the exact same thing.

@mfc, I do not know if you read this reply in this thread. Is your preference the same after reading this comment?

yeah, i agree with it, with preference for key over dongle (key is associated with security, dongle is associated with interoperability).

Debate in code

hadn't seen this, glad to see folks are attempting to implement it and further improve the readability of the Heads code.

[jans23]

I opt for @szszszsz 's suggestion, that Heads would use the exact product name (after initial enrolment). That would very precise and most easy to understand for all users.

Prior to the enrolment we would still need a generic term. Here I'm ok with both "dongle" and "key" as long as it differentiates to ordinary USB security dongles like Yubikey. The terms "Heads compatible..." and "OTP validating..." would both do the job. I agree with @tlaurion that "Heads" might go away in the future and also it may require more education of users.

[tlaurion]

@kylerankin : Heads implementation proposition that would fix Heads reproducibility issues when board config states CONFIG_LIBREMKEY for CI and users.

[tlaurion]

@MrChromebox As stated here, librem-hotp-verification is not reproducible even after #657 merged in.

[tlaurion]

@MrChromebox @kylerankin @szszszsz @jans23 : Is someone working on fixing Nitrokey/nitrokey-hotp-verification#13 (comment) ?

[MrChromebox]

@tlaurion I'll put this on my to-do for next week

[MrChromebox]

@tlaurion we can close this now, yes?

[tlaurion]

@tlaurion we can close this now, yes?

@MrChromebox : From preliminarily tests, libremkey-hotp-verification doesn't seem reproducible yet.

[MrChromebox]

@MrChromebox : From preliminarily tests, libremkey-hotp-verification doesn't seem reproducible yet.

hmm, what did I miss then?

[tlaurion]

Only observable public information for the moment for commit id ad84c38

• libremkey_hotp_verification produced binary:
  • 2b52542b6b9400f0e2c2964697fb9b5564c1ef1f29f7aaeaa56eb99cd8125ded (CircleCI)
  • e6bfb237ed9a89ea3b10d0a869c0bfe210acc8f11dcc5fdd621c39e189640b45 (GitlabCI)

Will redo another branch for both circleci and gitlabci to build against the same code, this time including CONFIG_LIBREMKEY in board config to see if same results are happening across different public CIs.

[tlaurion]

@MrChromebox @kylerankin @jans23 @osresearch : Builds for x230-libremkey.

Non-reproducibility reproduced:

• CircleCI:
  • 0f09faea0d7685768da1242eb63b46baaf2c8718deabcac2e87851fbda2d2e61 ./bin/libremkey_hotp_verification
• GitLabCI:
  • a4d0d8fef8b66f94d846b3a48ed781615be731cbe2bfaf8707591d0ad050c21e ./bin/libremkey_hotp_verification
• bzImage are also different, the rest is reproducible.

To be considered for Heads collaboration and future requirements:
All projects depending on Heads should have public CIs, so reproducibility issues are quickly discovered, troubleshooted and fixed (like this one and current bzImage non-reproducibility.)

Also, Heads's added security statement is based on reproducibility of its built ROMS, showing proof to users that the actual code hosted online actually produced flashed ROM for a specific commit id that is burned inside of their /etc/config.

If all modules are pinned to specific commit IDs/versions and those modules produce reproducible binaries included in a reproducible ROM, users can then rebuild that Heads commit id at any future time and should arrive at the same exact final ROM hash, while all included files hashes are included in downloadable artifacts online, permitting internal/external random audits.

Else, Heads added security in completely nullified/void, requiring the users to blindly trust ROM producers without any possibility of being able to reproduce any result themselves, requiring them to have faith in OEMs. ROM non-reproducibility is no better then what closed source OEM offers (backdoored?) with their binary-only versions of produced ROM images, flashed by manufacturers without source code or any possibility to audit easily without having a researcher's mindset.

It should be as easy for users as validating hashes internally for a commit id produced artifact hash file/reflashing original rom and seeing fashrom saying that it didn't change anything (after adding public key and config.user file prior to flash internally), or more thoroughly, externally, by dumping ROM, removing public key and config.user and comparing against downloaded reproducible ROM for a specific commit id for expected hash match.

Other then that, we otherwise also offer security by obscurity, which needs to be fixed at all cost if security of users are really a priority.

Todos:
This means attacking conjointly #571 so that a Heads mainsteamed GitLabCI/CircleCI/xyz configurations permit to build all boards on different build environments from different docker images.

This also means that we have to decide what we do with including IFD and neutered ME in #307.

[szszszsz]

I have problem with running the heads build at all. It stops both on the local machine (both custom Docker or gitlab-runner) and on the Gitlab on MAKE musl-cross. Are these artifacts cached on the main CI? Any ideas? Details:

• https://gitlab.com/szszszsz/heads/-/jobs/485364811

[BlackMaria]

@szszszsz you have an issue with your job...

Two suggestions:

1. go to the repo job page and increase the timeout, the issue is we are building everything before heads gets built, this may take up to 2h ( or even 4 on a slower system )
   • https://gitlab.com/szszszsz/heads/-/settings/ci_cd ( I suggest 2h minimum )
2. [optional] Increase the log length: Add this to your runners toml config output_limit = 102400

I can produce a working dockerfile that builds heads, if you find you are missing something ( iirc we now need texinfo to build something )

[tlaurion]

@BlackMaria

I can produce a working dockerfile that builds heads, if you find you are missing something ( iirc we now need texinfo to build something )

This is what was done originally by @osresearch and that I changed in my attempt to generalize caching and reproducibility with user exposed dependencies to make documentation coherent all at once and test things transparently. Of course it comes with the initial cost of building crossgcc for the first build of a branch but after that, the cache is reused, and since modules are now all pinned to specific commit ID, those changes should be picked by rebuilding to Heads newer commit.

Here are my two configs for review, improvement suggestions:
CircleCI
GitlabCI

@BlackMaria : for the runner timeout, I've got lost in Gitlab issue tracking pointing to fss branch and they seem to have lost themselves in the goal of fixing the runner timout value difrectly in the config. Can you advise otherwise? We need a CI ressource on Heads project, at least to start us in the goad direction

[szszszsz]

Below are my findings so far, copied from Nitrokey/nitrokey-hotp-verification#13 (comment).

Edit: @BlackMaria: thank you for the suggestion, will check it.

---

The initial change (setting the Release configuration) has helped, as the __FILE__ macro was used in the assertions, so the build path was part of the result. However there are still binary differences, which are not immediately obvious, e.g. the symbols for hidapi are moved exactly 0x1000 further in the binary while comparing the files with diffoscope.
So far I have not localized the direct cause for the current differences:

1. The reproducible build tool reprotest, which injects intentionally changes to the file system, build path, system clock and locale, have not found any differences, while executed on the same Docker images, but with distro's current GCC.
2. I have applied flags which are used in other projects built in heads project, but this has not changed the final sha256 sum in the local Docker tests. The used Hidapi library also receives the flags for stable build (confirmed with the compiler invocation), and is managed by the CMake.

This means the issue is occurring only during the Heads build, with that particular build-provided compiler configuration.
Further ideas:

• compare object files to limit the occurrence of the difference (e.g. with sha256), and compare the different ones with diffoscope (download them as artifacts from CI and run diff locally);
• run reprotest in the Heads build environment, with local compiler used;
• change build system from CMake/Make to simple Make or Meson/Ninja. Rationale is, that CMake-generated Makefile potentially introduces some instability. CMakefile itself is stable (as in: no random files order, or getting system variables, at least in the upstream unpatched version).

I will add in the further commit local build reproduction test with Docker and reprotest tool.

[tlaurion]

Fixed under #724 which was just merged upstream and built here ATM: https://app.circleci.com/pipelines/github/osresearch/heads/154/workflows/c0e760de-58c7-4315-a7ce-a45adbd75f31/jobs/161

Keeping open so I implement a gitlab-ci based on Fedora and push so that we confirm that binaries share same hashes.

[tlaurion]

I confirm that libremkey-hotp-verification:

CircleCI debian-11 :
f3bda451660100e9bf9c5c2eda760e31e2be5514d69dc20dd88248e0ea179566 ./bin/libremkey_hotp_verification

GitlabCI Fedora-30 :
f3bda451660100e9bf9c5c2eda760e31e2be5514d69dc20dd88248e0ea179566 ./bin/libremkey_hotp_verification