Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issues causing the Rustc process wrapper to be built non-determ… #2216

Merged
merged 1 commit into from
Oct 27, 2023

Conversation

UebelAndre
Copy link
Collaborator

@UebelAndre UebelAndre commented Oct 26, 2023

This change aims to ensure the process wrapper is built determinisitically, thus reducing churn in cases where multiple developers share a bazel remote cache. The shared cache is notable as the binaries can be deterministically produced for a single user for a single checkout as some host information is embedded in the compiled binaries. The host information is stored in the debug-info section of the executable. This can be resolved by stripping it. The other issue of determinism is rust_binary_without_process_wrapper does not resolve ${pwd} like the process wrapper does, thus stripping sandbox paths from rustc outputs. To solve for this a small bash and batch script was added to account for this behavior. The intent of these scripts is to use no host executables and instead rely on pure bash and batch to accomplish this and keep any actions which use this as hermetic as possible.

The following tables were produced on macOS. Note that rules_rust_2 is simply another checkout of rules_rust on the same commit

Before the changes in this PR:

path rules_rust sha256 rules_rust_2 sha256
bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib 60db194b3e98b67cc0702c6b6c48c5bc8fcf7d723f3ece6a7a24a53888158c7e 3eac0b443ba160e3a1bde3b023f4e953bb9fc9722e430b5ded6dbb723bc2b532
bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/util/process_wrapper/process_wrapper 32e7840602c977b76bba0bc39da768d35db3e7a17dbf96e455727be2b2f0151f 0d248279bbc9b17be5914b41a66759f939ef4da77f8a40a5ce6fa3bf339648ad

After the changes:

path rules_rust sha256 rules_rust_2 sha256
bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib 07432f5d207da854266ccde212243c67c29907f81a7619434ce9f608b1658d96 07432f5d207da854266ccde212243c67c29907f81a7619434ce9f608b1658d96
bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/util/process_wrapper/process_wrapper cbf17392338aabfc942d975f95a49a67b731c5e597a7d27e3d9cf4d4a06b8f2c cbf17392338aabfc942d975f95a49a67b731c5e597a7d27e3d9cf4d4a06b8f2c

closes #2092

@UebelAndre UebelAndre force-pushed the process_wrapper branch 10 times, most recently from a95d281 to 057fc49 Compare October 26, 2023 15:38
@UebelAndre UebelAndre marked this pull request as ready for review October 26, 2023 18:05
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unclear if this actually results in deterministic outputs. I do not have a windows machine to verify.

@UebelAndre UebelAndre mentioned this pull request Oct 27, 2023
@scentini
Copy link
Collaborator

I don't understand the windows code, but LGTM otherwise, and I think we can leave windows determinism to be confirmed by a windows user. Thanks!

@UebelAndre UebelAndre merged commit ec5fd40 into bazelbuild:main Oct 27, 2023
@UebelAndre UebelAndre deleted the process_wrapper branch October 27, 2023 12:20
ghost pushed a commit to sandbox-quantum/sandwich that referenced this pull request Nov 1, 2023
[`0.30.0`] fixes [a bug](bazelbuild/rules_rust#2216) that basically was breaking the hermeticity of Bazel

[`0.30.0`]: https://github.com/bazelbuild/rules_rust/releases/tag/0.30.0

GitOrigin-RevId: a9e00959c7a6be3fe86d54399f475444421d7450
ghost pushed a commit to sandbox-quantum/sandwich that referenced this pull request Nov 1, 2023
[`0.30.0`] fixes [a bug](bazelbuild/rules_rust#2216) that basically was breaking the hermeticity of Bazel

[`0.30.0`]: https://github.com/bazelbuild/rules_rust/releases/tag/0.30.0

GitOrigin-RevId: a9e00959c7a6be3fe86d54399f475444421d7450
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tinyjson compilation is not reproducible
2 participants