v0.2.1 appears not to be reproducible

[druimalban]

Hi there!

I'm having significant trouble building the v0.2.1 release in a reproducible manner.
I've attempted this on three different computers: a VM running Debian 8, a remote server running Scientific Linux 7, and a computer running Gentoo, with the Musl C library.
My first attempt at building was on the computer running Gentoo. I was unable to build the PCIUtils module for some reason (I believe this was an issue in #164 , which is still open - interestingly this isn't an issue on Debian or Fedora, so they must do something differently), so I used the config file with Flashrom, PCIUtils and Networking commented out (I didn't need this module):

BOARD=x230
CONFIG_CRYPTSETUP=y
CONFIG_FLASHROM=n
CONFIG_GPG=y
CONFIG_KEXEC=y
CONFIG_UTIL_LINUX=y
CONFIG_LVM2=y
CONFIG_MBEDTLS=y
CONFIG_PCIUTILS=n
CONFIG_POPT=y
CONFIG_QRENCODE=y
CONFIG_TMPTOTP=y
CONFIG_DROPBEAR=y
CONFIG_XEN=n
CONFIG_LINUX_USB=y
CONFIG_LINUX_E1000E=n
CONFIG_BOOTSCRIPT=/etc/keylime-init

On the Gentoo system, I get a sha256 hash of 70b834ccc1c2f15a09562e072630987be62f258358bfe1cb4be482e95b3430e4;
On the Debian 8 system, I get a sha256 hash of 0562c4427eca58b43ace7f184f625b5198bfb586dbf287b3b744a8442f86bfde;
On the remote server, I get a sha256 hash of b88535ce799a79ec5e99fb4cc87b8f8d62d333dc9b11497235e8eaef2a222cdd.

I am not sure what is causing it.
The resulting bzImage on the Gentoo system has a hash of 86cf04a12b92e86bcb19142cc47e03679b38567c48311ca2d45f8966690fff45;
The resulting bzImage on the Debian 8 system has a hash of f416fe3b4d3fa0841aa28744fe3876ce708e7bad9967b45aedefbb3395772e66;
The resulting bzImage on the remote server has a hash of
f416fe3b4d3fa0841aa28744fe3876ce708e7bad9967b45aedefbb3395772e66.
This seems to suggest to me that the remote server and the Debian 8 system are producing Linux in a reproducible manner. However, something else seems to be the problem here, perhaps to do with the way the initrd is constructed.

For what it's worth, I am able to produce the same hash for the x230-flash target on the remote server and Debian. I'm unable to do so on the Gentoo system because of the PCIUtils issue. I'm not sure how to solve that.

I'd also like to add something - I'm a little confused as to why you put the qemu.rom and x230.rom on the release page (to produce hashes of), when Xen is not reproducible across environments, to my understanding. I've attempted to build both in a reproducible manner and have only managed to reproduce the qemu.rom image on the Debian system. I'm unable to reproduce the x230.rom image from both the x230-moc.config and x230-qubes.config. I'm not sure whether this is an issue on my end, or with the build system.

Furthermore, I've found making CONFIG_LINUX_USB=n and CONFIG_LINUX_E1000E=n or y and y makes no difference.

Thanks in advance!

[jpouellet]

Any chance you could post the actual images you built so they can be diffoscoped?

[druimalban]

Apologies, I meant to attach the ROMs:
On the Gentoo system:
https://share.riseup.net/#inVHyw7brW4R5bfkDYzRnw
On the Debian system:
https://share.riseup.net/#pSFfAVXfe0nQRn8M4N-lbg
On the remote server:
https://share.riseup.net/#NMqsN7NAiONE2E3lNiNzMg

Just to make sure the build wasn't leaving things behind (to my knowledge), I ran make modules.clean before. However, it made no difference, and I got the same hashes.
https://share.riseup.net/#NMqsN7NAiONE2E3lNiNzMg
Thanks again!

[osresearch]

Heads has patches to make xen reproducible; on the test platforms (ubuntu 16.04, fedora 25, debian 8) it seems to be built correctly.

Is there an easily accessible gentoo digitalocean droplet or aws image?

[osresearch]

The configuration options are for modules in the initrd, not in the bzImage, so they shouldn't affect your bzImage hash (but should affect the x230.rom hash).

[druimalban]

There are some images for the 'experimental' archive for Gentoo with Musl here: https://gentoo.osuosl.org/experimental/amd64/musl/
I used the 'hardened' variant, so the image 'stage4-amd64-hardened-musl-cloud-latest.tar.bz2' is perhaps most useful.
I used the testing packages (keywords ~amd64 in /etc/portage/make.conf), which has the newest toolchain (GCC 6). However, if the issue is building Heads on a Musl system, then it might not be necessary to do use the testing packages.

[osresearch]

I think I figured out the pciutil issue -- can you try this patch? #164 (comment)

[druimalban]

The PCIUtils issue has been solved, which is good.
However, I have now attempted to build config/x230-flash.config on the Gentoo system. I get the following sha256 hash: fff8fb909878b030de848d3bbfdb1e6f4d643ce0f285e3af63c1939fd6e75521
The ROM image is attached here:
https://share.riseup.net/#Q9-2ZM5o2EtaYplIZhTigA

[druimalban]

I couldn't get Hardened Gentoo up and running as a VM in Qubes (it should be possible, but I didn't have time to figure it out), so I built in a chroot - I was able to build fff8fb909878b030de848d3bbfdb1e6f4d643ce0f285e3af63c1939fd6e75521 in the chroot. This is good, because it seems to be reproducible within certain environments, I think, even if not across environments.

This suggests several candidates for the cause of the image building differently, albeit in a reproducible manner.

1. The modifications Hardened Gentoo make to the compiler might be causing it - . I'm not sure how this might affect the final product. These are to do with PIE and the SSP. I'm quite skeptical of this, however. It shouldn't affect the code produced by the compiler that is built.
2. Musl as the C library. Perhaps linking against certain libraries has unpredictable results. I think this is most likely.
3. Something else that Hardened Gentoo do in order to enable Musl. Many packages are patched in order to work with Musl as the C standard library.

I'm going to test this against other systems that use Musl, like Void or Alpine, in the next month. That might ascertain whether the problem is Musl, as Void and Alpine don't seem to do as much hardening of the compilers as Hardened Gentoo.

This has got me thinking - what happens when we try to build Heads on Mac OS? It shouldn't be a problem - my understanding is that Coreboot does build on Mac OS. It would be interesting to ascertain whether it builds reproducible across different environments. I'm aware that there are issues building Coreboot on BSD, however.

[tlaurion]

As of right now only #734 is problematic