update GnuPG and friends to 2.4.0

[saper]

gnupg 2.2.21 -> 2.4.0
libgcrypt 1.8.6 -> 1.10.1
libassuan 2.5.3 -> 2.5.5
libksba 1.4.0 -> 1.6.3
libgpg-error 1.46

[tlaurion]

@saper: took your PR and pushed it on my github to be built by CircleCI at https://app.circleci.com/pipelines/github/tlaurion/heads/1561/workflows/933c2b4e-c532-4166-bec5-2be8ca9166f2 (CircleCI error).

[tlaurion]

@saper fails at https://app.circleci.com/pipelines/github/tlaurion/heads/1561/workflows/933c2b4e-c532-4166-bec5-2be8ca9166f2/jobs/19454?invite=true#step-103-1904

[tlaurion]

GPGRT_CONFIG is not found in libgcrypt configure script...
Related but not related https://dev.gnupg.org/T5365

[saper]

afaik gpgrt-config is not strictly necessary. Looks like CI runs with a different compilation flags (-Werror ?), since I got the linker warning but everything works. We are actively removing rpath from binaries so it kind of does not wonder me. But probably this can be solved nicely.

[tlaurion]

@saper This is the issue where solution is not working
https://dev.gnupg.org/T6257

Related patch proposed is to be tested, not merged, upstream (and doesn't fix the issue at stake):
https://dev.gnupg.org/rCe2c0920fd7dea6e254badd12958982c66080b7db
which superseded but is the same as
https://dev.gnupg.org/rE628cb1a786bd5d420be17c2e9ffd407f4fbf517e
Both referred per above issue.

[tlaurion]

We rely on pkg-config to provide proper paths, where as of now, libgpg-error.so is found, but libgpg-error.so.1 is searched in wrong path and fails linking, both locally and under CircleCI (debian-11 based for build reproducibility (not to be confounded with reproducibility of builds. We just want a setup to recommend that works for all, in the meantime of having a build stack that produces same binaries).

I was trying to fix this and point you again to gpg-card binary addition (new from 2.3+) which needs to be explored to hopefully remove the screen scraping, that as you seen, is fragile and does not survive version bumps (and is a maintainership nightmare long term).

[tlaurion]

Updated #1350 (comment)

[tlaurion]

@saper fails at https://app.circleci.com/pipelines/github/tlaurion/heads/1561/workflows/933c2b4e-c532-4166-bec5-2be8ca9166f2/jobs/19454?invite=true#step-103-1904

@saper if you had a previous version of libgpgerror.so where searched for, that is why it works in your build env but that lucky build cannot be made again anywhere else.

I wiped install/x86, had to rebuild complaining modules (eg: make BOARD=x230-hotp-maximized libgpg-error.clean) to reproduce same thing that CircleCI complains about. Make install statements of modules copy things under install/x86 which are then reused by modules that depend on them.

CircleCI on the other hand builds clean (3 levels of cache) where only coreboot buildstack was reused (musl-cross-make cache + build/x86/coreboot-) since modules/ definition changed, so the biggest cache could not be reused.

Biggest cache is the one containing build/x86/* and install/x86 cache, so basically building "clean" (with muslc-cross cache and coreboot builds cache).

[tlaurion]

https://app.circleci.com/pipelines/github/tlaurion/heads?branch=gnupg-2.4.0-update
Will try to talk to gniibe

[tlaurion]

Seems like rhe issue are with tests not reusing same toolchain and libraries used for compilation. Getting rid of the faulty tests under libgcrypt might be the solution here

[saper]

I have disconnected the tests from the libgcrypt build with 924f78f

[saper]

Force pushed the last commit with a whitespace fix by @tlaurion

[tlaurion]

Note that ppc64 related patches are gone, will test those in next batch of tests, hopefully I will remember.

[tlaurion]

Some notes on size changes (produced by playing with initrd.cpio.xz under /tmp dir), comparing output of
xz -d initrd.cpio.xz ; cpio -i < initrd.cpio ; find . -type f -ls | sort -r -n -k7 |grep -v cpio | awk -F " " {'print $NF " " $7'} | sort > output

--- master-hotp-fbwhiptail/output	2023-03-27 14:02:18.647000000 -0400
+++ gpg24-hotp-fbwhiptail/output	2023-03-27 14:05:20.344000000 -0400
@@ -18,9 +18,12 @@
 ./bin/flash.sh 6653
 ./bin/flashtool 18336
 ./bin/generic-init 1345
-./bin/gpg 751456
-./bin/gpg-agent 314584
+./bin/gpg 828136
+./bin/gpg-agent 343984
+./bin/gpg-card 205704
+./bin/gpg-connect-agent 119216
 ./bin/gpg-gui.sh 9056
+./bin/gpgsm 442792
 ./bin/gpgv 150
 ./bin/gui-init 21584
 ./bin/hotp 5912
@@ -53,7 +56,7 @@
 ./bin/qrenc 5904
 ./bin/qubes-measure-luks 597
 ./bin/reboot 375
-./bin/scdaemon 346376
+./bin/scdaemon 450792
 ./bin/scp 22664
 ./bin/seal-hotpkey 3328
 ./bin/seal-totp 1545
@@ -72,7 +75,7 @@
 ./bin/whiptail 35
 ./bin/xx30-flash.init 639
 ./etc/ash_functions 1930
-./etc/config 625
+./etc/config 771
 ./etc/distro/keys/archlinux.key 1725
 ./etc/distro/keys/qubes-4.key 1629
 ./etc/distro/keys/qubes-testing.key 3078
@@ -95,11 +98,11 @@
 ./lib/libcryptsetup.so.12 403152
 ./lib/libc.so 596544
 ./lib/libdevmapper.so.1.02 284184
-./lib/libgcrypt.so.20 1054216
-./lib/libgpg-error.so.0 109520
+./lib/libgcrypt.so.20 776328
+./lib/libgpg-error.so.0 121704
 ./lib/libjson-c.so.5 81024
-./lib/libksba.so.8 199016
-./lib/libmbedcrypto.so.0 288272
+./lib/libksba.so.8 207216
+./lib/libmbedcrypto.so.0 325104
 ./lib/libnpth.so.0 14552
 ./lib/libpci.so.3 52272
 ./lib/libpci.so.3.5.4 52272
@@ -109,15 +112,18 @@
 ./lib/libqrencode.so.3 39232
 ./lib/libtpm.so 314728
 ./lib/libusb-0.1.so.4 18440
-./lib/libusb-1.0.so.0 84584
+./lib/libusb-1.0.so.0 92808
 ./lib/libuuid.so.1 14720
 ./lib/libz.so.1 108832
-./lib/modules/e1000e.ko 304272
-./lib/modules/ehci-hcd.ko 64800
-./lib/modules/ehci-pci.ko 10728
-./lib/modules/usb-storage.ko 132360
-./lib/modules/xhci-hcd.ko 159840
-./lib/modules/xhci-pci.ko 10568
+./lib/modules/e1000.ko 223136
+./lib/modules/ehci-hcd.ko 116912
+./lib/modules/ehci-pci.ko 12312
+./lib/modules/ohci-hcd.ko 73280
+./lib/modules/ohci-pci.ko 13896
+./lib/modules/uhci-hcd.ko 61320
+./lib/modules/usb-storage.ko 140888
+./lib/modules/xhci-hcd.ko 276096
+./lib/modules/xhci-pci.ko 13448
 ./mount-boot 1375
 ./output 0
 ./run/cryptsetup/.placeholder 0

[tlaurion]

Also note that neither gpg-connect-agent, gpgsm nor gpg-card are needed, but sizes above show respective added space consomption outside of initrd.cpio.xz being compressed (which can be obtained by comparing coreboot stitching output in logs of CircleCI output, and where addition of those 3 breaks builds of legacy boards and t520 board as of now).

I would advise removing gpg-connect-agent as well unless really necessary.
Screen scraping haven't changed that much, where next step on my side will be change functions under oem-factory-reset and separate them to more properly call key-attr changes, define type of key desired etc, where first step and goal on my side is prototype this to be useable to recreate current RSA 3072 keysize without breaking anything and do the same in ram and then key to card, and move from there.

[saper]

Sure, we can drop gpg-connect-agent and gpgsm for now, if we use them only for testing.

[tlaurion]

Suppressed things further for gpg under https://github.com/tlaurion/heads/tree/gnupg-2.4.0-update

[tlaurion]

Work in progress, smartcards do not seem to work

@saper what do you mean?

[tlaurion]

@saper

Also note that the following would be needed for the factory reset/reownership wizard to work with 2.4

tlaurion@207c8b5

@@ -130,12 +128,12 @@ gpg_key_reset()
echo ${ADMIN_PIN_DEF}
echo ${USER_PIN_DEF}
echo 0
- echo y
echo ${GPG_USER_NAME}
echo ${GPG_USER_MAIL}
echo ${GPG_USER_COMMENT}
+ echo ${USER_PIN_DEF}

[tlaurion]

Note that ppc64 related patches are gone, will test those in next batch of tests, hopefully I will remember.

@saper damnit I forgot.
Any reason why that patch was removed?

[tlaurion]

Basically: why this patch has been removed for ppc64 @saper?

[saper]

No reason, I probably made a mistake and though this has already been included upstream.

[tlaurion]

If you have a quick answer on that it would be nice, otherwise will need to test.

[tlaurion]

To be tested on top of #1381

[saper]

I wouldn't hold my breath :(

[tlaurion]

We should revert adding tools/gpg-connect-agent

[tlaurion]

We should revert removing --disable-debug?
--disable-doc can be enabled

[tlaurion]

Could add
--disable-libdns

[tlaurion]

2.4.1 fixes issues with copy to card for yubikey.
2.4.2 is out.

[tlaurion]

Merged under #1422 with minor modification