rules_rust: enable pipelined compilation. #1275
Conversation
|
Mentioning @jsgf since you wrote tips on how to implement this in #228 (comment) based on Buck v2. |
gigaroby
force-pushed
the
pipeline-bzl
branch
9 times, most recently
from
f35f8f7
to
8f0c279
Compare
gigaroby
force-pushed
the
pipeline-bzl
branch
2 times, most recently
from
f668ad9
to
9273c86
Compare
gigaroby
force-pushed
the
pipeline-bzl
branch
11 times, most recently
from
d750801
to
ea39bfe
Compare
gigaroby
force-pushed
the
pipeline-bzl
branch
3 times, most recently
from
8cfe57e
to
b462a7b
Compare
gigaroby
force-pushed
the
pipeline-bzl
branch
3 times, most recently
from
b40a00d
to
f4eaa71
Compare
|
@gigaroby I'm curious if you have already used this on a decently sized project? If so, did you see any noticeable difference? |
Building the Rust toolchain using bazel this seems to reduce the wait from 2h to 40 minutes. It's an enormous win! |
|
Nice! Looking forward to test this on our repo. |
What are you building? A a full rust toolchain? So rustc and things? Or do I misunderstand that. |
Yes, a full Rust toolchain including LLVM, bootstrapped from an existing beta rustc and clang. |
|
Nice! @scentini is it possible to cut a release with this feature soon? |
|
@durin42 Presumably it had no effect on the llvm bits of the build? |
|
@jsgf not unless those parts of LLVM are in rust themselves. This only unlocks parallelism for rlib -> rlib arches in the build graph. |
The recent support for pipelined compilation (bazelbuild#1275) is great. There are some situations, however, where we need to disable pipelining for specific crates, e.g. bazelbuild#1584. This PR adds a crate annotation option to disable pipelining for rust_library targets generated by cargo_bazel.
The recent support for pipelined compilation (bazelbuild#1275) is great. There are some situations, however, where we need to disable pipelining for specific crates, e.g. bazelbuild#1584. This PR adds a crate annotation option to disable pipelining for rust_library targets generated by cargo_bazel.
The recent support for pipelined compilation (bazelbuild#1275) is great. There are some situations, however, where we need to disable pipelining for specific crates, e.g. bazelbuild#1584. This PR adds a crate annotation option to disable pipelining for rust_library targets generated by cargo_bazel.
The recent support for pipelined compilation (bazelbuild#1275) is great. There are some situations, however, where we need to disable pipelining for specific crates, e.g. bazelbuild#1584. This PR adds a crate annotation option to disable pipelining for rust_library targets generated by cargo_bazel.
The recent support for pipelined compilation (#1275) is great. There are some situations, however, where we need to disable pipelining for specific crates, e.g. #1584. This PR adds a crate annotation option to disable pipelining for rust_library targets generated by cargo_bazel. The alternatives would be patching the crate with the annotations system or disabling pipelining globally.
Pipelined compilation allows better parallelism during builds as it
allows libraries to generate lightweight metadata files to unlock other
depencies. These metadata files (.rmeta) can only be used to unlock
library -> library dependencies and do not affect builds in any other
way. This is currently the default in cargo:
https://internals.rust-lang.org/t/evaluating-pipelined-rustc-compilation/10199