oem-factory-reset fails to generate PGP keys with some Yubikeys

[icequbes1]

Issue:
Upon flashing the most recent heads master (commit e492786, Dec 1 2021) on a T430 machine built with make BOARD=t430 and all settings reset, PGP key generation always failed for at first-boot with a whiptail error displaying "GPG Key automatic keygen failed". The user is then left to generate PGP keys manually and provision heads on their own as detailed on the wiki 1. Heads works fine after.

This issue describes debugging and identified workarounds and serves to discuss steps forward prior to submission of new Pull Requests.

Investigation:
Debugging the failure scenario showed that the prompts supplied by oem-factory-reset during the gpg --card-edit "generate" command were not what gpg was expecting, primarily due to Admin and User PINs.

PR #1063 was opened to fix the issue, assuming it was a regression that had not been caught. Before the Pull Request, Admin PIN followed immediately by User PIN was supplied by oem-factory-reset near the beginning of the prompts. The PR change kept User PIN at the beginning, but moved the Admin PIN to the end.

Further analysis showed success depended on which OpenPGP card was being used. In my case, a Yubikey 5 with firmware version 5.1.2 was used. No problems were seen with Nitrokeys before the PR was submitted.

The same discrepancy in gpg prompts were seen outside of the heads environment with the same Yubikey, further hinting at Yubikey being the issue.

In the end, it was found that the Yubikey appears to use an invalid value for an OpenPGP Data Object (PW1 Status), technically violating the OpenPGP spec. This mismatch causes gpg to execute different logic when authenticating PINs.

Until gpg performed an explicit action to set PW1 Status to a certain value within the OpenPGP spec, PW1 Status retained an invalid value, that, when evaluated as a boolean is interpreted to mean PW1 (User PIN) is valid for several commands. This is the behavior gpg seeks when performing a generate command, and once it sees a Nitrokey does not have this value set, it preemptively sets it early in the procedure, which requires an Admin PIN.

Workarounds, solutions, next steps:

1. Leave code as-is, document this possibility and encourage use of manual provisioning
2. Leave code as-is, document a "hack" that will make the Yubikey respond similar to a Nitrokey
   • Run admin, forcesig, [admin pin], iif Signature PIN shows as "not forced" in gpg --card-status
3. Make modifications to the code to detect which prompts are asked and respond appropriately

Considering I am the only one who has complained, I would think step 1 is most suitable and step 2 for those who like to tinker.

I don't see the benefit of option 3 if not many people have complained, especially if the identified alternatives in option 1 or 2 are suitable.

Thoughts on simply pursuing the first option? I note that oem-factory-reset is not even documented in the Wiki, so the work here is essentially just documenting oem-factory-reset and caveats (the Yubikey).

---

Detailed debugging:

Enabling --debug ipc when invoking gpg showed the reason for the discrepancy:

• On my Yubikey, after a factory reset, the OpenPGP Data Object "PW1 Status bytes" (tag 0xC4) value was "255". PW1 Status Bytes is a 2-bit field indicating if PW1 (User PIN) is valid for a single command (0b00) or several (0b01). This value is shown as "SCD GETATTR CHV-STATUS" by gpg.
• On other implementations, this value was "0" after factory reset.

GnuPG, upon seeing a value of "0" for PW1 Status bytes performs a scdaemon command to set the value to 0b01, so that the user is not continually prompted for the User PIN during the session. However, setting the value to 1 requires PW3 Authentication, or the Admin PIN.

This explains why Admin PIN is requested with the Nitrokey, but is not requested with a Yubikey.

Finally, once gpg sends the SCD GENKEY command to scdaemon, the Yubikey now requires the Admin PIN as PW3 is required to be verified for the OpenPGP GENERATE ASYMMETRIC KEY PAIR command. In the Nitrokey case, the Admin PIN has already been supplied.

It is also observed that peforming a factory-reset command within gpg or the gpg-connect-agent-based reset sending raw APDUs did not affect the value of PW1 Status. My belief is this value is potentially uninitialized before being explicitly set. Once explicitly set, it retains its value, though even after factory resets. It is unknown if the OpenPGP implementation even cares of this value.

The PW1 Status DO can be explicitly set to 0b00 with gpg --card-edit by running:

gpg/card> admin
gpg/card> forcesig

If the output of the gpg --card-status indicates:

Signature PIN ....: forced

Then, the card is in a good state and will now complete oem-factory-reset just like a Nitrokey. Signature PIN "forced" equates to NOT PW1_Status.

In the Yubikey, a PW1 Status of 255 (or even a valid value of 0b01) evaluates to false, which will show the Signature PIN as "not forced".

In comparison, a Nitrokey indicated "forced", meaning GnuPG will change it to "not forced" or cached, and oem-factory-reset will be happy as it is answering prompts under the assumption that the card does not cache PW1 (User PIN).

[tlaurion]

@icequbes1 to me the good solution is to bump gpg toolstack to 2.3+ as stated there: #1063 (comment) which makes gpg-card command available, which is a frontend to scdaemon, which is actually issuing commands to gpg card and would permit, as I understood it, gpg-card --batch usage (permitting to feed all needed paramaters at once and let scdaemon deal with standards without us having to care in proper orders of parameters like we do now). gpg --card-edit is forever having bad UX reputation.

So we expect there some uniformisation of USB Security dongles without a need to detect OpenGPG used versions after inclusion of that new built binaries under the version bumped of the modules/gpg2 toolstack requirement https://github.com/tlaurion/heads/blob/f9bac36b4180158f85e2d1180601c44a56441dc0/modules/gpg2#L54

The current approach of redirecting fd and status is, let's be frank, kinda hacky and should be avoided.
Otherwise, the magic could be enhanced so that we parse the questions asked (status) and feed them dynamically to fd.
This was considered static and working for all until you raised the issue in PR #1063.

If you have willingness/capacity to port gpg toolstack from 2.2.21 LTS to gpg 2.3+, you can see what worked and was needed to do in the past, including the reproducibility hacks provided in patches/* for all the gpg toolstack dependencies under https://github.com/osresearch/heads/pull/860/files. I was happy to see LTS postponed the problem to 2024 unless major discovered security issue. But if this is the easiest way to make this work for all, then maybe bumping to gpg2 2.3 is a good idea and modify oem-factory-reset. Note that the scope for oem-factory-reset is clearly for OEMs, not intended for end-users.... So the solution might be to push even more for manual provisioning here. Maybe it is time to have a gpg-factory-reset once gpg2 2.3 is in... My thoughts on oem-factory-reset are here: #1067 (comment)

A bump in gpg toolstack means taking all relative new versions of toolstack from modules/* Makefiles files, making sure they compile, making sure then can be encompassed into smallest board configs (currently legacy boards: eg t430-hotp-verification inclusion) and make sure that the hashes.txt of produced binaries matches locally and remotely, otherwise raising a reproducibility issue.

Meanwhile, I agree that manual keygen should probably be pushed forward more for YubiKeys @icequbes1, while this issue should be seen by Yubikey users when seeing the problem. The real solution, to me, lies under #771, personally.
Tag me if you particulate in current discussions!

[tlaurion]

Heads doesn't have ways to collect use cases and current usage of code base, but it seems that a lot of end users are using oem-fctory-reset code, which was never aimed at user reownership @jans23 @kylerankins @icequbes1 @MrChromebox

That needs to change for security goals. We should merge efforts here. How do we proceed?

[tlaurion]

#771 needs collaboration else security benefits are totally vain.

[jans23]

If OEM Factory Reset should be used by non-OEMs, I suggest to changing the name to Factory Reset.

[icequbes1]

There are several issues and enhancements at hand in your message @tlaurion: 1. Issue: Yubikey OEM factory reset failing 2. Enhancement: Upgrade GnuPG toolstack to 2.3 to allow use of `gpg-card` which might bring uniformity amongst various smartcards 3. Enhancement: (#771) Generate keys within heads, import into smartcard This GitHub issue is attempting to focus on #1 with a short-term, immediate resolution for end-users attempting to do a factory reset with a Yubikey. I'd like to only focus on this for the time being so it can be considered resolved and supply assistance to any users who might experience this issue now. I'll work on wiki documentation to supply this assistance. But for the future, #2 and #3 sound like competing enhancements, with #3 being a desired long-term solution. If that is the ultimate long-term solution, is there any reason aside from "`oem-factory-reset` smartcard uniformity" pursuing a GnuPG toolstack upgrade to 2.3? As an end-user who builds and flashes heads myself on hardware I've always had, I consider myself the "OEM" so I don't get the designation of certain tools as being only for OEMs versus end-users. "OEM Factory Reset" is in the menu when booting the machine, and automatically run when no keyring exists, so I don't see why end-users (even if purchased from an actual OEM) wouldn't use it if they need to reset their machines. I agree with @jans23 that renaming to "Factory Reset" could be more appropriate. However, the long-term direction of `oem-factory-reset` fits better in #771 as sub-tasks of the overall enhancement to generate keys within Heads and import to the smartcard. That issue can better address the goals and security concerns to ensure they're inline with the identified customer concern as well as adequate for end-users to understand if they need to reset their machines themselves.

[tlaurion]

For the sake of this bug report only, if I understand your debug trace correctly @icequbes1, it would be a matter of factory resetting, then doing a forcesig prior of continuing with actual prompts and answers to have the same behavior for Yubikey and Nitrokey/Librem Keys?

[icequbes1]

For the Yubikey, the hack is: 1. First boot system 2. Exit to recovery shell 3. gpg --card-edit 4. admin 5. forcesig 6. list (and verify Signature PIN says: forced) 7. Reboot the system 8. factory reset now works

[tlaurion]

@icequbes1 https://github.com/osresearch/heads/files/7637077/gpg_card_edit_error.txt
shared under #1063 (comment)

Shows that setting forcesig prior of continuing with other steps would be the simple fix.
The factory reset code worked at that step. But factory state differs between Nitro/Librem and Yubikey.

Then on Yubikey, running oem-factory-reset fails because of given debugging trace.

Sorry if I misunderstand you, but I still believe setting forcesig before going with actual code would work without additional changes.

[icequbes1]

The Yubikey needs to have a certain bit cleared before trying to do the oem-factory-reset in Heads. This can be done within Heads by exiting to recovery shell, or it can be done outside of Heads. The bit can be cleared with 'forcesig' command. The 'bit' that needs to be cleared has been observed to persist even after a Yubikey factory reset. Once the Yubikey has had the bit cleared, oem-factory-reset within Heads, as it currently exists on master, will then work without any changes to its code.

[MrChromebox]

any reason we can't detect if a Yubikey is present and then clear the bit accordingly before performing a factory reset?

[tlaurion]

@icequbes1 :
Why not simply put forcesig after this line?

https://github.com/osresearch/heads/blob/415a08a73259840e753fb1a4ca6cc3e99ed4b339/initrd/bin/oem-factory-reset#L83

Unless its a toggle...

[icequbes1]

@tlaurion, @MrChromebox Yes we can add logic to do it within oem-factory-reset. I haven't tested where exactly that logic would be performed, as using forcesig requires the Admin PIN and I'm not sure if that's cached by the daemon if there isn't a removal/re-insertion of the card. If it's cached, it would put us in the same spot of prompts differing; I'll run some tests.

[tlaurion]

So basically to have the same state to interact with between Nitrokey/Librem Key/Yubikeys, after doing the call for gpg --card-edit, admin, factory-reset https://github.com/osresearch/heads/blob/32e70316785f599e28c38c62a3ac72ba5f7acb27/initrd/bin/oem-factory-reset#L74-L81

• oem-factory-reset should check for output of gpg --card-status to make sure that Signature PIN is set to forced
• if not, since factory-reset sets the GPG Admin PIN to known value, just toggle it on to forced by calling gpg --card-edit, admin, forcesig
• proceed with actual code logic

The first following admin command will require PIN to be entered (local test here required to enter PIN to unset forcesig here, where going out of gpg back in didn't, but removing/reinserting device requested Admin PIN again).

So it is a logical path to think that first Admin command will require PIN to be entered and will cache it until device is disconnected. Rest of the commands there require PINs to be entered each time.

@icequbes1 ?

[tlaurion]

@githubuseravailable confirmed same issue on Yubikey 5 Nano here: linuxboot/heads-wiki#102 (comment), and that solution porposed in previous link fixed his problem.

Since we do not like to have issues constantly opened and closed for the same code issue, would either @icequbes1 or @githubuseravailable be willing to test a code fix if it was proposed through a PR for their used platform? (I repeat: I do not own Yubikeys and I do not want to answer repeating issues over time when code fix would prevent that.)

Please tag me here if willing to test code I would implement to fix reported and now confirmed issue. Troubleshooting an issue repeatedly and applying workaround manually is an undesirable thing to do. But testers owning hardware is required. Which is why we prefer when a working fix is proposed by hardware owners and where I could have tested to make sure no regression happens on what is owned and used by the rest of the community.

Thank you in advance for your collaboration and understanding. A bug should be fixed in code, not through workarounds.

[githubuseravailable]

willing to test a code fix if it was proposed through a PR for their used platform?

@tlaurion i can test the code fix, but do we need to re-compile & re-flash ?