Create automatic tests of Dasharo build documentation

[philipandag]

The problem you're addressing (if any)

Every historic release of Dasharo has to be possible to be built by anyone with access to the sources. The resulting firmware must be identical to the ones we publish. The process of building Dasharo for all supported devices is described at docs.dasharo.com, but it turns out the instructions are not always completely flawless.
#1139
#1143

The CI at https://github.com/Dasharo/coreboot/ makes sure the firmware can be build on all the devices, but it only checks it on the dasharo branch and does not do it according to the documentation, like a person would, but according to a CI script, which is not guaranteed to do the same steps as described in the documentation.

Describe the solution you'd like

An automatic test that will check if following the instructions from the documentation allows to build any historic release of Dasharo, for any device, which will be identical to the ones we publish.

The test should find all the published historic releases for every supported device, find the build documentations for the devices, build the firmware according to the instructions in the documentation and compare if the resulting binary is identical as the published release.

The build process should start from a fresh installation of a selected OS to make sure no dependency is missing from the documentation. Ideally from the https://github.com/dasharo/preseeds

Where is the value to a user, and who might that user be?

The ability to reproduce the historic releases, which is not always guaranteed due to faults in documentation

Describe alternatives you've considered

No response

Additional context

Discussion and a simple implementation of such test by copying the build instructions manually: Dasharo/open-source-firmware-validation#545

• Manual testing would be easy, but equally unreliable and time consuming as the reason for the issues with build documentation is human error. Multiple people have been creating and updating the build instructions over time. The instructions don't follow a strict pattern.
• The test must be performed by reading the documentation in some way. Copying the documentation to a script does not make sense as both of them would need to be kept up to date at the same time. And there still would be a lot of possible human errors involved.
• The documentation might need to be adjusted to be able to be analyzed by a bot. The .md or .html files could be analyzed to find out the build instructions. Either:
  • All the codeblocks should contain only valid commands, mindlessly following which would result in building the FW
  • Some well defined syntax should be used to identify the blocks to execute and/or places where things like the release revision has to be inserted.
• Using Jinja or another template language was suggested as a way of creating the build documentation to minimize the human error when creating it. The same templates could be used as an input to the test.
• Simplifying the build process with a universal build script was suggested as a way to decrease the human error and define the build process well. Such script would need to support all historical releases too. A trial at extending the build.sh script in coreboot repo to fulfill this task: Refactor build.sh to be usable as the standard way of building Dasharo coreboot coreboot#579. Using this script is not ideal as it is a part of the coreboot repository, which needs to be checked out to release revisions, which won't contain the script. If a build tool like that would be simple enough, so that the build process caould be performed in one command, the tests of the documentation, and the documentation as a whole could be simplified a lot.
• The build can be tested on qemu, although adding it to the CI of the documentation repository should be done carefully as building Dasharo so many times could easily take too long. One build in https://github.com/Dasharo/coreboot CI easily takes over 5 minutes.