Reproducible executable

[dinosaure]

Hi, I actually use cosmopolitan to make a program available here: https://bob.osau.re/
The program is built by our reproducible system for OCaml here: https://builds.osau.re/
The project which permits to use Cosmopolitan and OCaml is available here: https://github.com/dinosaure/esperanto

I recently upgrade esperanto to Cosmopolitan to 3.6.2 and my program is no longer reproducible (it used to be). It seems that the diff is towards the end of the generated binary. I've tried to reinforce some aspects for reproducibility (TIME and DATE) that weren't necessary before, but it doesn't change anything. Do you have any idea what changes have been made on Cosmopolitan to make Bob more reproducible?

I'm still not sure at what level reproducibility is broken. The change that makes Bob no longer reproducible at all is the update to 3.6.2 - in fact, I suspect a change at that level but I agree that the error could potentially be elsewhere.

[jart]

Right before the 3.6.2 release I wrote 32292ae which I believe improves build determinism, by ensuring the UNIX mode bits of ZIP artifacts can only be 0644 or 0755. This resolved issues where I was getting inconsistent builds on MacOS vs. Ubuntu because each OS had different default settings for how the group bits show up.

It would help me if you could explain what you mean by reproducible. What are you comparing? A binary before the release versus after? A binary built by cosmocc on one OS versus cosmocc on another? The bits at the end of the file are the ZIP artifacts. Since we know the issue is there, you could use the o//tool/decode/zip program included in cosmopolitan to dump out the zip headers as ASCII and then take a good hard look at the UNIX mode bits to see if it's related to my change.

[dinosaure]

Sorry for the delay, I'm currently presenting Bob and Cosmopolitan at ICFP 2024 and I still haven't made the slides!

A binary before the release versus after?

Yes, this is exactly the test to suit reproducibility. I've decompressed (with unzip - UnZip 6.00 of 20 April 2009, by Info-ZIP) the artifact and it seems that between each build, there's a difference in the symtabs. As an example, here's a .symtab.amd64 hexdump of two versions of bob.com:

00000000: 5359 4d54 0100 0000 0915 0000 0000 0000  SYMT............
00000010: 00d0 2100 0000 0000 00d0 2100 0000 0000  ..!.......!.....
00000020: 0000 4000 0000 0000 ffdf bd00 0000 0000  ..@.............
00000030: 4023 4307 f37f 0000 bc14 4707 f37f 0000  @#C.......G.....
...

00000000: 5359 4d54 0100 0000 0915 0000 0000 0000  SYMT............
00000010: 00d0 2100 0000 0000 00d0 2100 0000 0000  ..!.......!.....
00000020: 0000 4000 0000 0000 ffdf bd00 0000 0000  ..@.............
00000030: 4063 c506 407f 0000 bc54 c906 407f 0000  @c..@....T..@...
...

The difference is in the last line, but I don't know (yet) what it really is. There is also a difference in .symtab.arm64. Of course, this difference means that the huffman tree between the two versions is different and that the whole zip part of the two versions is therefore different.

[dinosaure]

I tried to understand a little more. It seems that this file format (according to SYMBOLS_MAGIC) is that of libc/runtime/symbols.internal.h. It seems that the diff is in the names field and the name_base field. At this stage, it's hard to say why these values change for each build, but I think this is the way to go. What do these values depend on?

[dinosaure]

It should be these lines:

cosmopolitan/libc/runtime/opensymboltable.greg.c

Lines 92 to 93 in 8f81451

	t->names = (uint32_t *)((char *)t + t->names_offset);
	t->name_base = (char *)((char *)t + t->name_base_offset);

I will try to re-unzip and see why we have a diff at this point but we don't on the rest of the .symtab.*.

[dinosaure]

My first impression is that these values are pointers. The first thing that comes to mind is ASLR, where the position of t is random. So you'd have to disable ASLR to compile a Cosmopolitan project?

[dinosaure]

I deactivated ASLR when I built my software and I can confirm that I can now reproduce it. So the error does concern the t pointer and ASLR (mmap). I'm not sure what you could do to ensure reproducibility with or without ASLR, as I don't have in-depth knowledge of Cosmopolitan. But a fix isn't necessarily urgent (or even necessary if it's explicitly stated that ASLR must be disabled to ensure reproducibility here). /cc @jart

[jart]

Wow that's some really impressive analysis. Thanks for putting all the work into figuring that out! See 41fc76c which fixes this. It'll be shipped in the next release hopefully this week.

[jart]

Also for what it's worth, I managed to spot another potential source of nondeterminism, which is now fixed. I don't know if you distribute .a libraries built by my ar.ape program, but if you do, then 1e9902a fixes an issue I noticed where it was copying the st_mode into the output. It'll now always set the mode to 0644 thus ensuring your users who attempt to reproduce your binary release artifacts from source, will be less likely to end up with a different checksum. That change will also allow me to avoid the recurrence of a build error I recently caused for @ahgamut's superconfigure project, caused by me introducing a new library, which was fixed in df04ab8.