Make pex output deterministic / reproducible

[stuhood]

To have deterministic / reproducible output, pex should produce a byte-for-byte identical output given identical inputs.

There are a few common cases where this breaks down:

1. unstable timestamps included in output archives (zip/tar files, generally)
   • This can be fixed by ensuring that pex uses hardcoded timestamps for the entries that it places in archives.
2. unstable shas/timestamps intentionally included in metadata
   • Fixing things like this might involve adding an option to either disable including this info, or to stabilize it.
3. unstable ordering of hash iteration between machines
   • Harder to hunt down, and harder to defend against. But fixing it involves using order preserving or sorted structures.
4. use of absolute paths, or paths that are host specific.

It is out of scope (for this ticket) to stabilize the input files to pex (ie, adding lockfile support). So in cases where the network is involved, structures should be sorted.

---

It's not clear which combination of these issues might be in play in pex, so it would be good to start by getting a reproducibility test/experiment harness in place that makes it easy to compare two pex outputs and identify the above issues.

[jsirois]

This will be an excellent property to have. We can break out issues linked here as we pick off individual bits. There may or may not need to be upstream work to support stability of the archives embedded in built pexes - namely exploded wheels in .deps/ that pex sometimes has to build via setup.py execution.

[Eric-Arellano]

Some initial thoughts on how to approach this.

• Use TDD. Add failing tests that we skip.
• Use Python stdlib's filecmp and maybe difflib modules. Former is like shell's cmp program, will compare byte by byte. Latter will help to print diffs for easier to understand tests.
• Acceptance tests:
  • pex -o 1.pex; pex -o 2.pex; cmp 1.pex 2.pex
  • pex requests cryptography -o 1.pex; pex requests cryptography -o 2.pex; cmp 1.pex 2.pex
    • The actual dependencies used is subject to change. Idea is to test a universal wheel, ABI wheel, and we should also test a wheel that gets built from sdist.
  • pex -m pydoc -o 1.pex; pex -m pydoc -o 2.pex; cmp 1.pex 2.pex
  • Test -c scenario. Haven't grokked what -c does yet, but seems important.
  • Specify --python and --python-shebang.
• Over time, identify which unit tests will be meaningful to include.
  • It's going to be very difficult to debug the acceptance tests, because Pex files are bytes.
  • Unit tests can make sure each of the important pieces, like the timestamp, shebang, etc, are reproducible.

Rather than first doing a complete audit, does a more whack-a-mole approach sound reasonable? Steps:

1. Add the failing acceptance tests.
2. Start by fixing one simple thing, like the timestamp. Add a unit test for this.
3. Keep going until acceptance tests pass.

Reason I'd prefer this approach is that I anticipate becoming much more familiar with Pex and reproducibility through actually modifying the code, rather than trying to study it at a high level and only then changing things after the audit.

[stuhood]

The whack-a-mole approach sounds reasonable, but I'm not sure that adding the failing acceptance tests at the beginning will be significantly more useful than doing some manual diffing of exploded pexes using something like:

ls -l $exploded_one > one.txt
ls -l $exploded_two > two.txt
diff -u3 one.txt two.txt

... there are diminishing returns on making the output of the acceptance tests insanely useful when they fail when it is so easy to inspect a failure using unix tools. But up to you!

[Eric-Arellano]

but I'm not sure that adding the failing acceptance tests at the beginning will be significantly more useful than doing some manual diffing of exploded pexes using something like:

Agreed. The intent of starting with an acceptance test in #717 is to:

1. Put to pen (i.e. code) the requirements of this project. Helps me to understand all the edge cases we need to handle.
2. Act as a milestone for my motivation. For example, my first goal is to get the base case of no args passing. Only then will I start to work on the 3rd party dependencies case.
   • This is just like our Py3 CI blacklist idea.
   • Once we get the base case going, we can temporarily comment out the cases that still fail so that we can avoid any regressions.

there are diminishing returns on making the output of the acceptance tests insanely useful

Also agreed. I think what you propose in #717 (comment) sounds reasonable, but we shouldn't go more than that in acceptance tests, especially because I think we will want unit tests to make regressions easy to debug.

[Eric-Arellano]

So first major issue: .pyc is not deterministic until Python 3.7 via PEP 552.

Only targeting Python 3.7+ is not acceptable because Pants must support 3.6 in CI.

The Pep proposes that the way forward is to simply not distribute .pyc. We seem to distribute the original source code for every file, so this is tenable, although it means losing the benefit of caching. We can default to shipping .pyc and only turn off with an env var or CLI flag at build time.

Note the Pep references python/cpython#296 as a way to get deterministic timestamps, but it looks like that was a PR to modify CPython itself that was rejected, and that we can't leverage its work. I can look more closely if we're interested, but it would dramatically increase our complexity.

[jsirois]

I think a flag to turn off precompile (already an option in the code of PEXBuilder.freeze et. al.) is the way to go. Better even might be a flag to turn on reproduceable builds as a whole that can serve as an umbrella for all the knobs you end up discovering we need.

[stuhood]

Agreed re: the umbrella flag.

[stuhood]

Not to move the goalposts too much (and perhaps this should even be a separate issue?), but: another level of reproducibility that is useful is the ability to ignore the timestamps on input files.

In particular, in the context of the remote execution API (usage-of-which I owe you a http://github.com/pantsbuild/pants/ ticket for; EDIT: pantsbuild/pants#7649), timestamps are not preserved: only names, contents, type, and the "execute" bit (no file ownership, no timestamps, etc). This means that if you run the same command twice with the remote execution API, the files that will be extracted into the workspace that pex will be running in will have different timestamps each time.

@Eric-Arellano : Having typed this out, I believe that fixing the issue of input timestamps would be fine as a followup to the remoting-in-pantsbuild-pants work (to further optimize a relatively rare case). @illicitonion : My feeling is that ignoring input timestamps primarily helps in the "recovering from a dirty node" case, rather than our ability to get cache hits in the first place... so it doesn't feel like it blocks remoting for tests. Would you agree?

[jsirois]

I think this is a non-issue, just a test proving my assertion is all that's needed. No part of PEX or the tools it uses looks at input file timestamps. Excepting:

• zipfile: this is being fixed anyhow
• pex/compiler.py: indirectly via py_compile, but since there will be no .pyc lying around in the compile case (which remote exec won't use anyway!) bytecode will always be compiled.