Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove package crate test --tools-pex-mismatch-warn option. #2

Open
jsirois opened this issue Dec 12, 2022 · 1 comment
Open

Remove package crate test --tools-pex-mismatch-warn option. #2

jsirois opened this issue Dec 12, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@jsirois
Copy link
Contributor

jsirois commented Dec 12, 2022

Our self-hosted Mac arm runner does not clean the work directory between runs like the GitHub hosted runners do; as such, the ~/.cache/pants/named_caches/pex_root, etc. sticks around. This exposes
a bug in Pants / Pex reproducibility when building PEXes where the ~/.cache/pants/named_caches/pex_root/installed_wheels/... that get packaged up into a PEX may (or may not) contain bytecode if the particular wheel was ever run against in the past. This leads to varying final PEX content.

Solve this issue in Pants or Pex - likely both. Pex likely needs to filter bytecode out of the installed_wheels dirs when building a PEX, and if this ends up being an option, Pants needs to plumb it or use it or both.

@jsirois
Copy link
Contributor Author

jsirois commented Dec 13, 2022

Confirming this is the issue.

For this CI run: https://github.com/pantsbuild/scie-pants/actions/runs/3679714653/jobs/6224494867#step:6:147

We get:

>> Verifying the tools.pex built by the package crate matches the tools.pex built by Pants
The tools.pex generated by Pants does not match ours:
Ours:  /Users/gha/actions-runner/_work/scie-pants/scie-pants/target/debug/build/package-8bbe6409b7e99a41/out/dist/tools.pex
->     83cd9a45f2b5cc1c5b68314c3c7ab048b9aeae0dd131a236d4a495a60a3781a8
Pants: /Users/gha/actions-runner/_work/scie-pants/scie-pants/dist/tools/tools.pex
->     3daa318b0d5c77b71591b014bf1b5ff28af44477e8a0aff649491e148b5b73d1

And I find the difference is entirely rogue .pyc files in the PEX generated by Pants.

gha@46375 ~ % diff <(zipinfo -1 /Users/gha/actions-runner/_work/scie-pants/scie-pants/target/debug/build/package-8bbe6409b7e99a41/out/dist/tools.pex) <(zipinfo -1 /Users/gha/actions-runner/_work/scie-pants/scie-pants/dist/tools/tools.pex)
302a303,307
> .deps/ansicolors-1.1.8-py2.py3-none-any.whl/colors/__pycache__/
> .deps/ansicolors-1.1.8-py2.py3-none-any.whl/colors/__pycache__/__init__.cpython-39.pyc
> .deps/ansicolors-1.1.8-py2.py3-none-any.whl/colors/__pycache__/colors.cpython-39.pyc
> .deps/ansicolors-1.1.8-py2.py3-none-any.whl/colors/__pycache__/csscolors.cpython-39.pyc
> .deps/ansicolors-1.1.8-py2.py3-none-any.whl/colors/__pycache__/version.cpython-39.pyc
gha@46375 ~ %

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant