metadata-link: add -Zmetadata-link flag #85793
Conversation
|
r? @varkor (rust-highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
This comment has been minimized.
This comment has been minimized.
| @@ -1037,7 +1037,8 @@ fn start_executing_work<B: ExtraBackendMethods>( | |||
| })); | |||
|
|
|||
| let ol = if tcx.sess.opts.debugging_opts.no_codegen | |||
| || !tcx.sess.opts.output_types.should_codegen() | |||
| || (!tcx.sess.opts.output_types.should_codegen() | |||
| && !tcx.sess.opts.debugging_opts.metadata_link) | |||
There was a problem hiding this comment.
This shouldn't be necessary. This is only relevant for the actual codegen, not for encoding metadata.
|
Can you clarify how this would work? Like how would be the sequence of rustc calls that a build system would use? It's not clear to me how it would run |
|
@ehuss It's exactly analogous to pipelined builds in Cargo, except that The goal is the shorten the critical path for the dependencies by using metadata-only builds, so only the final linking target (ie executable) is dependent on codegen in order to increase build parallelism. In this scenario we can assume there's unlimited computing resources, so we're trying to remove as coupling in the dependency graph as possible to unlock parallelism and reduce the overall build latency. So if you have the simple case of If we already have cached artifacts for any of these steps - say from a previous check build (assuming check uses -Zmetadata-link for this purpose) - then we can just skip the step and use the artifact from cache. (And conversely a check build can use metadata generated by one of these build steps.) (Ideally we could decouple the compilation of main from linking, but that's a whole other conversation.) I have a prototype of this in a build system, but I'm curious to see how well this works, especially with the decoupled caching for rlib and rmeta. |
That would be |
Yeah, as I said, a whole other conversation. Unless they've changed since I last looked at them, they're not (quite) suitable for what I have in mind. |
|
Wait a minute. Isn't this supposed to be equivalent to whaf |
|
@bjorn3 I'll check it out but it sounds like a roundabout way of getting the desired results. Edit: Hm, Edit2: Looking at |
|
Tests added. |
|
@bjorn3 Actually after further testing, I don't think Given |
|
Cc @hyd-dev who also looked into issues around |
This makes sure that building a crate with an rmeta dependency is viable as a linkable crate.
This adds the -Zmetadata-link flag, whose intent is to make the output of `--emit metadata` equivalent to `--emit metadata,link`. The goal is to allow pipelined builds with separate invocations of rustc. This is desireable for two reasons: 1. The "artifact notification" system that rustc and cargo use to communicate is very cargo-specific - it doesn't fit well into other builds systems such as Buck or Bazel. In general its incompatible with any build system which only recognizes output artifacts once the compiler process has terminated. This is a particular problem when compilation is distributed. 2. The rmeta file could be cached independently from the rlib. For example, if one generates compilation-ready rmeta files as part of "check" build, then those can be directly used for a full compilation from cache, without having to regenerate them. This means the build turns into a highly parallelizable invocation of `rustc --emit link` commands using cached `rmeta` files as input. Initially this flag `-Zmetadata-link`, but ultimately I'd expect to stabilize it as `-Cmetadata-link` (or a better name).
Make sure the resulting executable is 1) linkable, and 2) the same as built with `--emit metadata,link`.
Hm, that doesn't make sense. Something else is amiss. |
|
@bjorn3 OK, I've done some more experiments with
The fact that Edit: I just verified that |
|
I'm closing this in favour of using |
This adds the -Zmetadata-link flag, whose intent is to make the output
of
--emit metadataequivalent to--emit metadata,link. The goal isto allow pipelined builds with separate invocations of rustc. This is
desireable for two reasons:
The "artifact notification" system that rustc and cargo use to
communicate is very cargo-specific - it doesn't fit well into other
builds systems such as Buck or Bazel. In general its incompatible with
any build system which only recognizes output artifacts once the
compiler process has terminated. This is a particular problem when
compilation is distributed.
The rmeta file could be cached independently from the rlib. For
example, if one generates compilation-ready rmeta files as part
of "check" build, then those can be directly used for a full
compilation from cache, without having to regenerate them. This
means the build turns into a highly parallelizable invocation of
rustc --emit linkcommands using cachedrmetafiles as input.Initially this flag
-Zmetadata-link, but ultimately I'd expect tostabilize it as
-Cmetadata-link(or a better name).There's some outstanding problems/questions:
No tests. It's not clear to me that there are existing tests which cover the pipelinedAdded tests forbuild mode specifically;
src/tests/ui/rmetasomewhat touches on it.plain pipelined builds and with
-Zmetadata-link.The generated rmetas fromI'm not sure what I was seeing before, but this looks fine - they're within a couple of bytes of each other.--emit metadata -Zmetadata-linkare much largerthan those from
--emit metadata,linkand I don't know why. While they seem to workin local ad-hoc testing, I'm concerned they may have different performance/compilation time/something else
problems.
-Zalways-encode-mirbut I think they're logically distinct (eg I seethis new flag as being on stabilization track).
--emit metadata,link -Zno-codegenwhichsounds plausible but quite roundabout.
--emit metadataalways work this way, and maybeadd
--emit checkwhich emits a subset of rmeta for check builds. This would either meanthat there could be a mixture of different kinds of rmeta files which are usable for different uses
(ie, the confusion of two different
--emitoptions producing files with the same name but differentcontent. Alternatively
--emit checkcould generate (say).rcheckfiles, but that adds complexityto locating files, since we'd end up with a 3 level heirarchy of rlib > rmeta > rcheck in terms of
what the files are useful for. I experimented with this briefly, but its effects ended up being overwhelmingly
sprawling and complex (eg affecting Cargo as well).
-Zmetadata-linkis much more narrowly scoped and opt-in.