Maintain actual CI to confirm reproducibility of roms across distributions

[tlaurion]

Reproducible builds CI based on Ubuntu docker are not working for a long while now:

@osresearch
https://circleci.com/gh/osresearch/heads

@nemanjan00
https://gitlab.com/nemanjan00/heads-ci/

Does anyone have a working one?

Fedora-29 based one does work, but lacks hooks to build from master commits:
https://gitlab.com/tlaurion/heads

Any help appreciated in fixing any of those problems!

Sent from my Galaxy S3 using FastHub-Libre

[BlackMaria]

my $0.25. circleci probably has some restrictions or limits that we are hitting. I cant verify since I don't have access to the account.

1. Space: it may be related to the amount of disk being used.
2. Threads: tons of threads are spawned when using make -j 4. I would suggest removing the -j 4 and see what happens.
   • IMHO we don't really need it to build multi-thread anyway.

NB I have built master in a loop for a weekend and never hit any deadlocks etc.

[tlaurion]

@osresearch : that would be lovely if fixed.
Meanwhile, my gitlabci ubuntu builder works

[tlaurion]

An temporary fix for this was introduced by @osresearch here, using a specific docker image to have bootsrapping tools encompassed to make builds faster, while not permitting easy audit boostrapping dependencies upgrades that could break reproducibility.

While non-ideal, this works.

[tlaurion]

Good news, builds are thought reproducible again since 6b485ed
Edit: not. As per my own reproducibility tests of March 22 2020, it's still not attained completely (bzImage and hotp-verification still not being reproducible)

Any people experienced enough to propose better CircleCI and GitlabCI configurations to be in tree so that CircleCI and GitlabCI could automatically build for different hosts (docker images) and boards upon passed environment variables? (CircleCI doc)

• docker image (ex: ubuntu:18.04, defaulting to it)
  • depending on host image type (ex: ubuntu), install proper dependencies (apt, dnf, others...)
• board configuration (ex: BOARD=x230, defaulting to it)

The community could then subscribe to Heads project over CircleCI and automatically produce default downloadable board artifacts (x230, qemu) for default host environment (ubuntu), append board ROM sha256sum on osresearch's github release page(if built by osresearch), while other builders CIs could infirm/confirm reproducibility across other docker images for a same commit id+board in some way?

[tlaurion]

@osresearch : CI builds are failing again from CircleCI, since they changed their log retention policy, I guess. If you come up with a trick to only output failing logs from build/logs/* (only logs modified in the last 5 seconds shown on screen?)

[Gusher123]

An temporary fix for this was introduced by @osresearch here, using a specific docker image to have bootsrapping tools encompassed to make builds faster, while not permitting easy audit boostrapping dependencies upgrades that could break reproducibility.

While non-ideal, this works.

@tlaurion @osresearch This temporary fix seems to have broken the CI builds from Gitlab since the januari 15th commit 8e4b109 (musl-cross-make) to Heads.

[tlaurion]

@Gusher123: https://app.circleci.com/jobs/github/osresearch/heads/

[Gusher123]

Thanks. That url did not work for me, but https://app.circleci.com/pipelines/github/osresearch/heads did

[tlaurion]

@osresearch : We still have reproducibility issues on commit ad84c38 on those components:

coreboot image differs:

• 1f8405943fdaf6efea02cc07a7cd2f35020baab67d18712035586f5e50cbdbc3 (CircleCI)
• ef52c4eef87efa866e960c1e8f4531117a2d652a59245efdb8a3369777c26770 (GitlabCI)

Because of:

• bzImage :
  • 795b307945343c5c7eb0b55a9285bb73f96832d295da9be5e11887db804ebe02 (CircleCI)
  • 69961890c7ee425855f8620562b5627584f7b2c5f52132291836445087cc90ef (GitlabCI)
    Rest is reproducible. @osresearch ?

Other non-reproducible component:

• librem-hotp-verification (To be addressed in libremkey-hotp-verification is not reproducible #640 since not part of x230 or other generic, non CONFIG_LIBREMKEY board configs )
  Edit: comment here shows non-reproducibility.

Edit: librem-hotp-verification is reproducible again since 6916069

[tlaurion]

@osresearch
Another issue with the buildsystem happens when make versions mismatches on docker CI and cache is already being extracted.

Result is that make is built again, trying to reapply patches which fails.

EDIT: was issue #799 fixed with tlaurion@9d056a0

[tlaurion]

libremkey-hotp-verification issue was fixed under #722

[tlaurion]

Each commit now produces images on:
GitlabCI(Debian)
GitlabCI(Fedora-30) (Attaching my instance for automatic builds of master)

[tlaurion]

@osresearch @BlackMaria @MrChromebox @flammit :

First step accomplished:

• Each commit now triggers two CIs. For example: Click on the green check, and it will show you CircleCI and GitlabCI clickable builds.

• Make vmlinuz reproducible again.

  • 795b307945343c5c7eb0b55a9285bb73f96832d295da9be5e11887db804ebe02 /root/project/build/x230-hotp-verification/bzImage

  • 69961890c7ee425855f8620562b5627584f7b2c5f52132291836445087cc90ef /builds/tlaurion/heads-osresearch/build/x230-hotp-verification/bzImage

• Have heads.cpio reproducible

  • shared hash: 980a1e671df217b2fa68896dfc6920695a65ed02c4f7e8135c5f093f50731ed7 /builds/tlaurion/heads-osresearch/build/x230-hotp-verification/heads.cpio

• Have modules.cpio reproducible

  • shared hash: a2fe6993f933f9a99a8d2b999f400ae57f834bc6443679c45db879c1dcdf3222 /root/project/build/x230-hotp-verification/modules.cpio

• Have tools.cpio reproducible

• Have initrd.cpio.xz reproducible

• have reproducible roms

[tlaurion]

@BlackMaria : How would we make those build systems report their hashes to github for automatic comparison and reproducibility validation?

[tlaurion]

@szszszsz :
Update
osresearch and tlaurion CircleCIs now build on each commit most of the boards actively used by direct community members.

The above points are still missing, while CircleCI still misses a proper checksum statement to validate individual points:

• crossgcc module should be checksumed and build cache remade only when changed. fixed under CircleCI : specialized caches #798
• all other modules should be checksumed and build cache remade only when changed. fixed under CircleCI : specialized caches #798
• As of right now, the cache is static (undesired in the long run since not saved when changed)

I'm still struggling with GitlabCI with space missing and didn't found a proper way to fix it yet.

[tlaurion]

Becoming more urgent to produce ROM images for coreboot 4.12, as seen in #824 and consequentially #825 (comment)

@MrChromebox ? @flammit ?

[tlaurion]

I'm still struggling with GitlabCI with space missing and didn't found a proper way to fix it yet.

Fixed with https://github.com/tlaurion/heads/tree/GitlabCI_Fix
Now trying to use properly GitlabCI caching system #851

[tlaurion]

Next steps:

• Implement shellcheck on new scripts (maybe rename all shell scripts to .sh to ease finding them and testing them) (WIP: Shellcheck script files in initrd #862 )

• Respect pipeline logic in both CircleCI and GitlabCI, permitting ROM deployment (release)

• Have heads versioning tags ( Add git hash and tags for versioning #839 is a good start)

• Have fwupd (LVFS) uploaded from releases uploaded to github. (Depends on Add git hash and tags for versioning #839)

[Thrilleratplay]

@tlaurion

from #851

goal here is to be able to build all boards on the same code base and different build system, auditable with packages being downloaded and caching fastening the process from one build to the next.

That I understood. The mention of coreboot-4.12 is why I made the suggestion. Apparently there are quirks with automake and aclocal in addition to dependency changes. It may have been easier to start from there instead of through trial and error with the base Debian image.

Question here would be: can that docker image be used with current build system and what would be the advantages of it? That would remove main OS as main obstacle for origin build tools. But we cannot say that the project would enforce reproducible builds. We could then say that Heads uses coreboot toolset, and modify Makefile, coreboot to not rely on musl-cross-make, but rely solely on that docket image and have instructions for user to deploy docker on all OSes.

I am still not sure to what extent the ultimate goal is, if it is for anyone to download the Heads source and compile it expecting the result to be the same hash as the CI builds, I do not think this is possible. Reading through What's in a build environment? on reproducible-builds.org, Make board=xxx would need to ensure the session's variables name of the user running the build, locale, timezone, and other environmental variables are the same without interfering without altering the system wide variables as this could break other running processes on the system. The user name is the most difficult for a user without full admin privileges. Playing around with GUIX, I am not sure this would fulfill the requirement either as it is requires the user to install a new package manger with daemon, again requiring admin privileges, to Linux distros that may nor may not support it. Currently, it does not build using the Arch AUR package as one of the dependencies has a build error. Even if GUIX was supported on every Linux distro, it would not still exclude Windows and MacOS users from developing.

Although the reproducible-build docs recommend using VMs and containers, I am not saying using a docker container is the solution and am still need to experiment with CircleCI and GitlabCI.

[Thrilleratplay]

@tlaurion In addition to shellcheck, a further TODO may be bash script unit testing. I have been using shellspec in thrilleratplay/coreboot-builder-scripts. It isn't perfect but it is a lot better than the other bash testing frameworks I have tried.

[Thrilleratplay]

@tlaurion Just curious, has musl.cc been considered an alternative to compiling musl-cross-make. I can understand where it could be rejected as there is some deviation from musl-cross-make but I could not find where it was mentioned in any of the Github tickets.

[tlaurion]

@Thrilleratplay : The current considered alternatives, by more knowledgeable people then I am are:

1. coreboot-sdk (with issues on docker image reproducibility (quote needed) : Goal here would be to replace all documentation we have now so that docker could be used on any OS to build heads in a reproducible way. Sub-instructions could show how to reproduce the docker image itself and be reused, which would be a big plus so Heads don't have to maintain patches to undo upstream work (python2-> python3, gawk, make etc) as all upstream OSes upgrade their build toolstack which have proved to be problematic to even build a same Heads commit years later. If that docker image, specified in Heads for a specific Heads commit could reproduce the same rom output 5 years from now, I think the goal would be met. That would imply modifying main Makefile and coreboot patches for musl-cross so that the coreboot-sdk musl-cross be used directly, instead of building musl-cross, economizing precious time both on CI builds and users building only for a specific board. One question that rises from this approach, which is untested, is what would happen if the coreboot-sdk for 4.12 is used to build 4.81 roms. No idea.
2. guix-buildstack : This is another approach @daym is playing with. The goal here would be similar to docker above, while guix-toolstack would be an additional layer added on top of any (Linux) based OS layer/docker, permitting in build instructions (and CI) to use a specific guix-toolstack revision in the goal of building the same ROM output. As opposed to the docker image above, we could easily modify the guix-toolstack to fit our needs (let it be LTO-compliant compiler (where LLVM is not interesting yet, from what I've read, but I might be misinformed).
3. Current musl-cross-make approach: musl-cross-make is a "portable" alternative to musl-cross, meaning that what is found after building musl-cross-make and calling an install, which is under heads, under ./crossgcc, is supposed to be portable. For reproducible builds, it means less chances of having hardcoded paths since buildstack is in relative paths. The current approach permits having multiple coreboot version being build, where coreboot is instructed to build, per coreboot configs, to ANY_TOOLCHAIN. That may or may not be the viable solution as opposed to coreboot-sdk or guix-toolchain.
4. musl.cc: I have not invatigated nor understand the clear differences from the quick diagonal reading of 10 minutes I have made on the subject.

The main objective here would be to, IMHO, to distance ourselves the most possible from the host toolchain. For example, even if we use musl-cross-make right now and the abstraction is is supposed to imply, we are still facing kernel reproducibility issues (#734) opened for more then a year after having moved away of musl-cross for musl-cross-make. I am not knowledgeable enough to understand why those changes are coming from, unfortunately.

But my intuition is that 1 and 2 are the most desired approaches.

• 1 because readily available while limiting ourselves for LTO and other optimizations we might want
• 2 would resolve more easily LTO integration.
• 1 would be applicable on top of any OS
• 2 would require docker + guix-toolstack.

[Thrilleratplay]

@tlaurion Thank you for the detailed answer, it was more than I expected.

From my local testing, I can answer a few questions. The coreboot-sdk docker versions from at least the two years are not able to compile coreboot 4.8, only coreboot 4.12. Compiling heads on coreboot-sdk for x230 requires two additional packages; cpio and texinfo.

I was trying to add musl-cross-make already compiled but became lost trying to add an exception to the module to check the make install locations. As the musl-cross-make module build directory is 7.5G, adding that to the docker image does not seem reasonable. This is why I asked about musl.cc.

@daym showed me how to pull his guix-on-heads image and planned on trying it later today. I think it will require docker running --privlieged. This is a potential issue for some end users but I am more concerned with trying to run a privileged image on CircleCI.

[tlaurion]

Hurray, CircleCI is able to sustain cache for multiple linux and coreboot versions:

Found a cache from build 588 at heads-modules-and-patches-zGxbPlRSx6OtBxTpidFp2N538LPES9DQ4z+9sH4XnVA=<no value>
Size: 7.6 GiB
Cached paths:
  * /root/project/packages
  * /root/project/crossgcc
  * /root/project/build

Downloading cache archive...

From #867 built https://app.circleci.com/pipelines/github/tlaurion/heads/541/workflows/9d19e4e0-a6ab-4d68-bfa1-f47d61b9b760/jobs/588 and reused https://app.circleci.com/pipelines/github/tlaurion/heads/541/workflows/b191e704-769e-4742-a3af-12fd0a4b7cfe/jobs/589 for the same modules and patches applied.

Now if someone has ideas on having the same musl-cross-make cache being downloaded on different docker images to test reproducibility and push to github releases on tags, that would be the next step in the goal of pushing Heads releases to LVFS!

[tlaurion]

I was trying to add musl-cross-make already compiled but became lost trying to add an exception to the module to check the make install locations. As the musl-cross-make module build directory is 7.5G, adding that to the docker image does not seem reasonable. This is why I asked about musl.cc.

@Thrilleratplay we see in past CircleCI cache that building a cache, compressing and uploading cache takes a while (16m45), while downloading it is really fast (4m21s). Building all boards without a modules cache takes 3h15m. So each time we touch a module/patchset, rebuilding all boards takes a while but actually produce a cache that then shortens that process to under 30 minutes. This is really big improvement.

Building a docker image could be done the same way a cahce is built, when buildstack changes (musl-cross-make or guix-toolstack dependencies) just like in cache.

When rebuilding the same modules/patches, we can see here that the build time for each board is reduces at maximum, where it could still be optimized since not everything is considered in the cache right now. For exemple, make and gawk are not cached inside of the musl-cross-make cache, which should probably be added. We see in the first board in line that is being built a build time of 9m22 where subsequent boards builds are just plainly reusing cache (41s, 1m23 etc). So as long as the same modules patches are used (which is normally the case under Heads projects, most of the PR are affecting scripts changes, not mudules nor patches) there won't be any need of rebuilding boards from a clean state, and at least musl-cross-make/guix-toolstack cache would be reused, where at best, the cache for built modules is reused, which now permits to build all boards over CircleCI for a commit under 27minutes!

My belief here is that it might be useless to try to create docker images, where caches could definitely be used for that purposes. If we go the path of guix-toolstack, that would be the same principles as understood, where it could be built from scratch (to test reproducibility where guaranteed) or downloaded as a layer prior of simply building boards, with cross compiler settings set in Makefile and other modules being adjusted to use that toolchain.

@daym @Thrilleratplay @MrChromebox @osresearch: any advice welcome prior of switching build toolstack or going one direction more then the other. Please read #571 (comment)

[Thrilleratplay]

@tlaurion FYI, when trying to get #872 to build, I found that CircleCI config version 2.1 and version 2 have a lot of incompatibilities but could not find a good document outlining how to migrate between them. As one of these how store-artifacts behaves and is called and that you spent so much time with this, I decided to use shellcheck in Debian. Be aware of this before updating the config version.

[tlaurion]

@Thrilleratplay : haaaaaa. Will have to check, I had no idea.

[tlaurion]

@tlaurion FYI, when trying to get #872 to build, I found that CircleCI config version 2.1 and version 2 have a lot of incompatibilities but could not find a good document outlining how to migrate between them. As one of these how store-artifacts behaves and is called and that you spent so much time with this, I decided to use shellcheck in Debian. Be aware of this before updating the config version.

@Thrilleratplay what incompatibilities have you found?

[Thrilleratplay]

@tlaurion I stopped at store-articles but here is a last build error I had with CircleCI version 2.1. I think in version 2.1 is was renamed store_articles

[tlaurion]

Seems like local cli validator might be of help here, while 2.1 is in test right now. Interesting what can be done with orbs.
Not a priority though :)

[tlaurion]

@daym your input on improvements and #571 (comment) would be more then welcome!

[tlaurion]

@daym showed me how to pull his guix-on-heads image and planned on trying it later today. I think it will require docker running --privlieged. This is a potential issue for some end users but I am more concerned with trying to run a privileged image on CircleCI.

@Thrilleratplay @daym This is not the case anymore? How would we replace musl-cross-make here to depend on guix-toolstack from docker? This is exciting.

[Thrilleratplay]

@Thrilleratplay @daym This is not the case anymore? How would we replace musl-cross-make here to depend on guix-toolstack from docker? This is exciting.

@tlaurion Sorry, at the time I was not 100% sure of this and did not want to provide incorrect information. After becoming more comfortable with Guix in Docker, I can confirm that the --privilieged flag is not required when running Guix package manager so long as it is run with the --disable-chroot flag in the container.

I will post my current plan in #927.

[tlaurion]

I think this is fixed by #1661