rustc: Stabilize options for pipelined compilation #62766

alexcrichton · 2019-07-17T21:13:34Z

This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in #60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).

The technical changes performed here were to stabilize the functionality
proposed in #60419 and #60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:

A new --json flag was added to the compiler.
The --json flag can be passed multiple times.
The value of the --json flag is a comma-separated list of
directives.
The --json flag cannot be combined with --color
The --json flag must be combined with --error-format=json
The acceptable list of directives to --json are:
- diagnostic-short - the rendered field of diagnostics will have a
  "short" rendering matching --error-format=short
- diagnostic-rendered-ansi - the rendered field of diagnostics
  will be colorized with ansi color codes embedded in the string field
- artifacts - JSON blobs will be emitted for artifacts being emitted
  by the compiler

The unstable -Z emit-artifact-notifications and --json-rendered
flags have also been removed during this commit as well.

Closes #60419
Closes #60987
Closes #60988

This commit enables pipelined compilation by default in Cargo now that the requisite support has been stablized in rust-lang/rust#62766. This involved minor updates in a number of locations here and there, but nothing of meat has changed from the original implementation (just tweaks to how rustc is called).

alexcrichton · 2019-07-17T21:15:36Z

It's also worth pointing out that the two tracking issues (#60419 and #60987) are not done with FCP, so this shouldn't be merged until they're done. I hoped to get this up early though to start feedback sooner.

jsgf · 2019-07-18T08:06:55Z

Are diagnostic-short and diagnostic-rendered-ansi exclusive?

src/bootstrap/test.rs

eddyb · 2019-07-18T14:05:58Z

After mentioning my preference for "just adding to the JSON format without an opt-in", I have to say that I like this design! Using -J instead of --json might also be interesting, but that is likely pushing it too far.

alexcrichton · 2019-07-18T14:27:37Z

@jsgf no they can be combined and both affect how the rendered field turns up.

ehuss · 2019-07-21T19:20:39Z

Since this makes the rendered field either "short" or "human", I just want to bring up the cache-messages feature again. I'm still a little concerned about the issues I raised earlier. When swapping between formats, the cache may replay the wrong format.

Does the following proposal sound reasonable? I think the best option is where cargo and rustc share a rendering library. However, since it will likely be a very long time before that is done, perhaps we can plan to stabilize cache-messages with limited "short" support. It would cache whatever format was originally requested (human or short). Then when replaying, it would replay whatever was cached. This might be a little confusing if the user is switching between formats, but I suspect this should be extremely rare if ever happens.

I'm happy with that plan, I just want to confirm that it sounds reasonable.

Centril · 2019-07-21T22:37:29Z

@alexcrichton When adding relnotes, please remember to also add the appropriate milestone (1.38 in this case). :)

alexcrichton · 2019-07-22T13:48:13Z

@ehuss oh sorry to clarify I haven't forgotten about that I just figured that it wasn't something we were going to solve right out the gate (or would rather require further changes to the compiler to fix).

I don't think what you mention is entirely unreasonable since switching between formats is likely rare, but in some sense rustc/cargo share at least part of "rendering library" by agreeing on what ansi color codes are. That still requires rustc to give Cargo exact text though which is a bit of a bummer.

In any case though I think that this should at least be forwards compatible with a scheme where cargo/rustc agree on a short rendering scheme and Cargo can replay both messages given one invocation of rustc, and otherwise I just wanted to be sure to not couple the stabilizations too closely.

alexcrichton · 2019-07-24T21:13:20Z

I suspect this may be outside of @estebank's wheelhouse as well, so I'll otherwise do...

r? @eddyb

This commit stabilizes options in the compiler necessary for Cargo to enable "pipelined compilation" by default. The concept of pipelined compilation, how it's implemented, and what it means for rustc are documented in rust-lang#60988. This PR is coupled with a PR against Cargo (rust-lang/cargo#7143) which updates Cargo's support for pipelined compliation to rustc, and also enables support by default in Cargo. (note that the Cargo PR cannot land until this one against rustc lands). The technical changes performed here were to stabilize the functionality proposed in rust-lang#60419 and rust-lang#60987, the underlying pieces to enable pipelined compilation support in Cargo. The issues have had some discussion during stabilization, but the newly stabilized surface area here is: * A new `--json` flag was added to the compiler. * The `--json` flag can be passed multiple times. * The value of the `--json` flag is a comma-separated list of directives. * The `--json` flag cannot be combined with `--color` * The `--json` flag must be combined with `--error-format=json` * The acceptable list of directives to `--json` are: * `diagnostic-short` - the `rendered` field of diagnostics will have a "short" rendering matching `--error-format=short` * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics will be colorized with ansi color codes embedded in the string field * `artifacts` - JSON blobs will be emitted for artifacts being emitted by the compiler The unstable `-Z emit-artifact-notifications` and `--json-rendered` flags have also been removed during this commit as well. Closes rust-lang#60419 Closes rust-lang#60987 Closes rust-lang#60988

alexcrichton · 2019-07-29T15:18:11Z

Gonna send a ping to @rust-lang/compiler for this as well, it'd be great to get this in for the 1.38 release and it will take some time to land this, land the subsequent Cargo support, and then finally land the Cargo update in rust-lang/rust.

oli-obk · 2019-07-30T06:19:41Z

@bors r+

bors · 2019-07-30T06:19:43Z

📌 Commit 1731233 has been approved by oli-obk

bors · 2019-07-30T08:39:40Z

⌛ Testing commit 1731233 with merge f690098...

…r=oli-obk rustc: Stabilize options for pipelined compilation This commit stabilizes options in the compiler necessary for Cargo to enable "pipelined compilation" by default. The concept of pipelined compilation, how it's implemented, and what it means for rustc are documented in #60988. This PR is coupled with a PR against Cargo (rust-lang/cargo#7143) which updates Cargo's support for pipelined compliation to rustc, and also enables support by default in Cargo. (note that the Cargo PR cannot land until this one against rustc lands). The technical changes performed here were to stabilize the functionality proposed in #60419 and #60987, the underlying pieces to enable pipelined compilation support in Cargo. The issues have had some discussion during stabilization, but the newly stabilized surface area here is: * A new `--json` flag was added to the compiler. * The `--json` flag can be passed multiple times. * The value of the `--json` flag is a comma-separated list of directives. * The `--json` flag cannot be combined with `--color` * The `--json` flag must be combined with `--error-format=json` * The acceptable list of directives to `--json` are: * `diagnostic-short` - the `rendered` field of diagnostics will have a "short" rendering matching `--error-format=short` * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics will be colorized with ansi color codes embedded in the string field * `artifacts` - JSON blobs will be emitted for artifacts being emitted by the compiler The unstable `-Z emit-artifact-notifications` and `--json-rendered` flags have also been removed during this commit as well. Closes #60419 Closes #60987 Closes #60988

bors · 2019-07-30T12:26:21Z

☀️ Test successful - checks-azure
Approved by: oli-obk
Pushing f690098 to master...

alexcrichton · 2019-07-30T14:29:58Z

Thanks @oli-obk!

Arnavion · 2019-07-31T05:49:35Z

@alexcrichton I guess nightly-2019-07-30 got released without the cargo change, as will future nightlies until cargo is updated? Right now compiling anything with pipelining = true in cargo config fails with error: Unrecognized option: 'json-rendered'. I guess we have to stop using pipelining until the cargo update is pulled in?

alexcrichton · 2019-07-31T14:12:42Z

That is correct, yes. Pipelining was always a nightly-only feature and this is the breakage that entails. The Cargo update, also mentioned in the description of this PR, is at rust-lang/cargo#7143.

Arnavion · 2019-07-31T17:22:10Z

Cool. Thanks for confirming.

This commit enables pipelined compilation by default in Cargo now that the requisite support has been stablized in rust-lang/rust#62766. This involved minor updates in a number of locations here and there, but nothing of meat has changed from the original implementation (just tweaks to how rustc is called).

…=ehuss Enable pipelined compilation by default This commit enables pipelined compilation by default in Cargo now that the requisite support has been stablized in rust-lang/rust#62766. This involved minor updates in a number of locations here and there, but nothing of meat has changed from the original implementation (just tweaks to how rustc is called).

Run Clippy without json-rendered flag Removed in rust-lang#62766 Replacing it with `--json=diagnostic-rendered-ansi` fails: ``` error: using `--json` requires also using `--error-format=json` ``` Running `./x.py clippy src/libstd` locally works fine (with colors) on Linux so I don't know if there is something to fix.

Pkgsrc changes: * Adapt to the move of the implementation of random numbers. * Remove patch which is no longer relevant (Signals.inc) * Cross-build currently fails due to the still unresolved rust-lang/rust#62558, so bootstrap kits for 1.38.0 have to be built natively, and will follow shortly. * Bump bootstrap requirements to 1.37.0 except for armv7-unknown-netbsd-eabihf which I've neither managed to cross-build nor build natively. Upstream changes: Version 1.38.0 (2019-09-26) ========================== Language -------- - [The `#[global_allocator]` attribute can now be used in submodules.][62735] - [The `#[deprecated]` attribute can now be used on macros.][62042] Compiler -------- - [Added pipelined compilation support to `rustc`.][62766] This will improve compilation times in some cases. For further information please refer to the [_"Evaluating pipelined rustc compilation"_][pipeline-internals] thread. - [Added tier 3\* support for the `aarch64-uwp-windows-msvc`, `i686-uwp-windows-gnu`, `i686-uwp-windows-msvc`, `x86_64-uwp-windows-gnu`, and `x86_64-uwp-windows-msvc` targets.][60260] - [Added tier 3 support for the `armv7-unknown-linux-gnueabi` and `armv7-unknown-linux-musleabi` targets.][63107] - [Added tier 3 support for the `hexagon-unknown-linux-musl` target.][62814] - [Added tier 3 support for the `riscv32i-unknown-none-elf` target.][62784] \* Refer to Rust's [platform support page][forge-platform-support] for more information on Rust's tiered platform support. Libraries --------- - [`ascii::EscapeDefault` now implements `Clone` and `Display`.][63421] - [Derive macros for prelude traits (e.g. `Clone`, `Debug`, `Hash`) are now available at the same path as the trait.][63056] (e.g. The `Clone` derive macro is available at `std::clone::Clone`). This also makes all built-in macros available in `std`/`core` root. e.g. `std::include_bytes!`. - [`str::Chars` now implements `Debug`.][63000] - [`slice::{concat, connect, join}` now accepts `&[T]` in addition to `&T`.][62528] - [`*const T` and `*mut T` now implement `marker::Unpin`.][62583] - [`Arc<[T]>` and `Rc<[T]>` now implement `FromIterator<T>`.][61953] - [Added euclidean remainder and division operations (`div_euclid`, `rem_euclid`) to all numeric primitives.][61884] Additionally `checked`, `overflowing`, and `wrapping` versions are available for all integer primitives. - [`thread::AccessError` now implements `Clone`, `Copy`, `Eq`, `Error`, and `PartialEq`.][61491] - [`iter::{StepBy, Peekable, Take}` now implement `DoubleEndedIterator`.][61457] Stabilized APIs --------------- - [`<*const T>::cast`] - [`<*mut T>::cast`] - [`Duration::as_secs_f32`] - [`Duration::as_secs_f64`] - [`Duration::div_duration_f32`] - [`Duration::div_duration_f64`] - [`Duration::div_f32`] - [`Duration::div_f64`] - [`Duration::from_secs_f32`] - [`Duration::from_secs_f64`] - [`Duration::mul_f32`] - [`Duration::mul_f64`] - [`any::type_name`] Cargo ----- - [Added pipelined compilation support to `cargo`.][cargo/7143] - [You can now pass the `--features` option multiple times to enable multiple features.][cargo/7084] Misc ---- - [`rustc` will now warn about some incorrect uses of `mem::{uninitialized, zeroed}` that are known to cause undefined behaviour.][63346] Compatibility Notes ------------------- - Unfortunately the [`x86_64-unknown-uefi` platform can not be built][62785] with rustc 1.39.0. - The [`armv7-unknown-linux-gnueabihf` platform is also known to have issues][62896] for certain crates such as libc. [60260]: rust-lang/rust#60260 [61457]: rust-lang/rust#61457 [61491]: rust-lang/rust#61491 [61884]: rust-lang/rust#61884 [61953]: rust-lang/rust#61953 [62042]: rust-lang/rust#62042 [62528]: rust-lang/rust#62528 [62583]: rust-lang/rust#62583 [62735]: rust-lang/rust#62735 [62766]: rust-lang/rust#62766 [62784]: rust-lang/rust#62784 [62785]: rust-lang/rust#62785 [62814]: rust-lang/rust#62814 [62896]: rust-lang/rust#62896 [63000]: rust-lang/rust#63000 [63056]: rust-lang/rust#63056 [63107]: rust-lang/rust#63107 [63346]: rust-lang/rust#63346 [63421]: rust-lang/rust#63421 [cargo/7084]: rust-lang/cargo#7084 [cargo/7143]: rust-lang/cargo#7143 [`<*const T>::cast`]: https://doc.rust-lang.org/std/primitive.pointer.html#method.cast [`<*mut T>::cast`]: https://doc.rust-lang.org/std/primitive.pointer.html#method.cast [`Duration::as_secs_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.as_secs_f32 [`Duration::as_secs_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.as_secs_f64 [`Duration::div_duration_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.div_duration_f32 [`Duration::div_duration_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.div_duration_f64 [`Duration::div_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.div_f32 [`Duration::div_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.div_f64 [`Duration::from_secs_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.from_secs_f32 [`Duration::from_secs_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.from_secs_f64 [`Duration::mul_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.mul_f32 [`Duration::mul_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.mul_f64 [`any::type_name`]: https://doc.rust-lang.org/std/any/fn.type_name.html [forge-platform-support]: https://forge.rust-lang.org/platform-support.html [pipeline-internals]: https://internals.rust-lang.org/t/evaluating-pipelined-rustc-compilation/10199

rust-highfive assigned estebank Jul 17, 2019

This was referenced Jul 17, 2019

Enable pipelined compilation by default rust-lang/cargo#7143

Merged

Tracking issue for stable -C emit-artifact-notifications rustc flag #60419

Closed

Tracking issue for unstable --json-rendered flag #60987

Closed

alexcrichton mentioned this pull request Jul 17, 2019

Tracking issue for stabilizing "pipelined compilation" #60988

Closed

7 tasks

drrlvn reviewed Jul 18, 2019

View reviewed changes

src/bootstrap/test.rs Outdated Show resolved Hide resolved

alexcrichton force-pushed the stabilize-pipelined-compilation branch from 3906a1f to 2e6a064 Compare July 18, 2019 15:13

This comment has been minimized.

Sign in to view

alexcrichton mentioned this pull request Jul 19, 2019

LinuxTools builders fails to (mabye?) push at the end of PR builds #62811

Closed

Centril added this to the 1.38 milestone Jul 21, 2019

This comment has been minimized.

Sign in to view

alexcrichton force-pushed the stabilize-pipelined-compilation branch from 2e6a064 to a769554 Compare July 24, 2019 21:12

rust-highfive assigned eddyb and unassigned estebank Jul 24, 2019

This comment has been minimized.

Sign in to view

alexcrichton force-pushed the stabilize-pipelined-compilation branch from a769554 to 2ad1335 Compare July 24, 2019 21:46

This comment has been minimized.

Sign in to view

alexcrichton force-pushed the stabilize-pipelined-compilation branch from 2ad1335 to 1731233 Compare July 26, 2019 14:47

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 30, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 30, 2019

bors mentioned this pull request Jul 30, 2019

Stabilise disable minification #56359

Closed

bors merged commit 1731233 into rust-lang:master Jul 30, 2019

alexcrichton deleted the stabilize-pipelined-compilation branch July 30, 2019 14:29

alexcrichton mentioned this pull request Jul 31, 2019

Remove derives Encodable/Decodable and unstabilize attribute #[bench] #62507

Merged

mati865 mentioned this pull request Aug 21, 2019

Run Clippy without json-rendered flag #63781

Merged

chrissimpkins mentioned this pull request Feb 9, 2020

ICE from associated const in field type of lifetime-parametric struct #68875

Closed

Wulf mentioned this pull request Jan 31, 2023

rendered with terminal colours oli-obk/cargo_metadata#217

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Stabilize options for pipelined compilation #62766

rustc: Stabilize options for pipelined compilation #62766

alexcrichton commented Jul 17, 2019

alexcrichton commented Jul 17, 2019

jsgf commented Jul 18, 2019

eddyb commented Jul 18, 2019

alexcrichton commented Jul 18, 2019

This comment has been minimized.

ehuss commented Jul 21, 2019

Centril commented Jul 21, 2019

alexcrichton commented Jul 22, 2019

This comment has been minimized.

alexcrichton commented Jul 24, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

alexcrichton commented Jul 29, 2019

oli-obk commented Jul 30, 2019

bors commented Jul 30, 2019

bors commented Jul 30, 2019

bors commented Jul 30, 2019

alexcrichton commented Jul 30, 2019

Arnavion commented Jul 31, 2019

alexcrichton commented Jul 31, 2019

Arnavion commented Jul 31, 2019

rustc: Stabilize options for pipelined compilation #62766

rustc: Stabilize options for pipelined compilation #62766

Conversation

alexcrichton commented Jul 17, 2019

alexcrichton commented Jul 17, 2019

jsgf commented Jul 18, 2019

eddyb commented Jul 18, 2019

alexcrichton commented Jul 18, 2019

This comment has been minimized.

ehuss commented Jul 21, 2019

Centril commented Jul 21, 2019

alexcrichton commented Jul 22, 2019

This comment has been minimized.

alexcrichton commented Jul 24, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

alexcrichton commented Jul 29, 2019

oli-obk commented Jul 30, 2019

bors commented Jul 30, 2019

bors commented Jul 30, 2019

bors commented Jul 30, 2019

alexcrichton commented Jul 30, 2019

Arnavion commented Jul 31, 2019

alexcrichton commented Jul 31, 2019

Arnavion commented Jul 31, 2019