enable parallel rustc front end in nightly builds #117435

SparrowLii · 2023-10-31T08:48:28Z

Refers to the MCP, this pr does:

Enable the parallel front end in nightly builds, and keep the default number of threads as 1. Then users can use the parallel rustc front end via -Z threads=n option.
Set it up to serial front end for beta/stable builds via bootstrap.
Switch over the alt builders from parallel rustc to serial, so we have artifacts without parallel to test against the artifacts with parallel.

cc @cjgillot @nnethercote @bjorn3 @Kobzol

rustbot · 2023-10-31T08:48:39Z

This PR modifies src/bootstrap/src/core/config. If appropriate, please also update CONFIG_CHANGE_HISTORY in src/bootstrap/src/lib.rs and change-id in config.example.toml.

rustbot · 2023-10-31T08:52:30Z

This PR changes config.example.toml. If appropriate, please also update CONFIG_CHANGE_HISTORY in src/bootstrap/src/lib.rs and change-id in config.example.toml.

src/bootstrap/src/core/config/config.rs

oli-obk

I'm gonna let people chime in on this for a day or two. If I don't approve by Thursday myself, r=me

lqd · 2023-11-01T14:52:46Z

Do we need to land #115220 prior to this PR? Or the fact that it applies to more than a single thread is enough not to (in my mind people are immediately going to try >1 threads).

oli-obk · 2023-11-01T15:25:51Z

More than one thread is deadlock prone and anyone opting into it with the -Z flag has to accept that I think

SparrowLii · 2023-11-03T03:05:33Z

@oli-obk I think we can merge now

@bors r=oli-obk rollup=never

bors · 2023-11-03T03:05:40Z

📌 Commit 248dd14 has been approved by oli-obk

It is now in the queue for this repository.

src/ci/run.sh

matthiaskrgr · 2023-11-03T11:43:30Z

@bors rollup=never

bors · 2023-11-03T14:40:31Z

⌛ Testing commit 248dd14 with merge bb4d1c3...

enable parallel rustc front end in nightly builds Refers to the [MCP](rust-lang/compiler-team#681), this pr does: 1. Enable the parallel front end in nightly builds, and keep the default number of threads as 1. Then users can use the parallel rustc front end via -Z threads=n option. 2. Set it up to serial front end for beta/stable builds via bootstrap. 3. Switch over the alt builders from parallel rustc to serial, so we have artifacts without parallel to test against the artifacts with parallel. r? `@oli-obk` cc `@cjgillot` `@nnethercote` `@bjorn3` `@Kobzol`

davidtwco · 2023-11-06T05:15:00Z

r? @davidtwco

@bors r=oli-obk,davidtwco

bors · 2023-11-06T05:15:02Z

📌 Commit f2a40e9 has been approved by oli-obk,davidtwco

It is now in the queue for this repository.

bors · 2023-11-06T07:41:26Z

⌛ Testing commit f2a40e9 with merge f9b6446...

bors · 2023-11-06T09:46:33Z

☀️ Test successful - checks-actions
Approved by: oli-obk,davidtwco
Pushing f9b6446 to master...

rust-timer · 2023-11-06T21:31:33Z

Finished benchmarking commit (f9b6446): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.9%	[0.4%, 5.1%]	209
Regressions ❌ (secondary)	2.7%	[0.4%, 9.1%]	219
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.9%	[0.4%, 5.1%]	209

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.2%	[0.4%, 2.6%]	28
Regressions ❌ (secondary)	2.5%	[0.5%, 6.8%]	61
Improvements ✅ (primary)	-1.6%	[-2.2%, -0.9%]	2
Improvements ✅ (secondary)	-2.2%	[-2.5%, -2.0%]	3
All ❌✅ (primary)	1.0%	[-2.2%, 2.6%]	30

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.5%	[0.4%, 5.2%]	186
Regressions ❌ (secondary)	3.3%	[1.3%, 8.3%]	119
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.5%	[0.4%, 5.2%]	186

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 636.284s -> 661.435s (3.95%)
Artifact size: 304.53 MiB -> 308.99 MiB (1.46%)

89: Automated pull from upstream `master` r=Dajamante a=github-actions[bot] This PR pulls the following changes from the upstream repository: * rust-lang/rust#117006 * rust-lang/rust#117511 * rust-lang/rust#117641 * rust-lang/rust#117637 * rust-lang/rust#117631 * rust-lang/rust#117516 * rust-lang/rust#117190 * rust-lang/rust#117292 * rust-lang/rust#117603 * rust-lang/rust#116988 * rust-lang/rust#117630 * rust-lang/rust#117615 * rust-lang/rust#117613 * rust-lang/rust#117592 * rust-lang/rust#117578 * rust-lang/rust#117435 * rust-lang/rust#117607 90: bump serde and serde_derive r=tshepang a=Dajamante Trying to get around the failure seen in #86 Co-authored-by: Ralf Jung <post@ralfj.de> Co-authored-by: Esteban Küber <esteban@kuber.com.ar> Co-authored-by: SparrowLii <liyuan179@huawei.com> Co-authored-by: Matthias Krüger <matthias.krueger@famsik.de> Co-authored-by: Gurinder Singh <frederick.the.fool@gmail.com> Co-authored-by: Michael Goulet <michael@errs.io> Co-authored-by: Thom Chiovoloni <thom@shift.click> Co-authored-by: klensy <klensy@users.noreply.github.com> Co-authored-by: Jack Huey <31162821+jackh726@users.noreply.github.com> Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com> Co-authored-by: hkalbasi <hamidrezakalbasi@protonmail.com> Co-authored-by: bors <bors@rust-lang.org> Co-authored-by: Sven Marnach <sven@mozilla.com> Co-authored-by: Rémy Rakic <remy.rakic+github@gmail.com> Co-authored-by: aissata <aimaiga2@gmail.com>

pnkfelix · 2023-11-07T18:02:16Z

I am not sure that wg-parallel-rustc was anticipating there being this much of a regression here.
asked for followup in zulip for wg-parallel-rustc
not marking as triaged

SparrowLii · 2023-11-08T00:59:16Z

@pnkfelix Sorry for not explaining in the MCP. We decided in the working group meeting early this year that wall-time is a better choice to measuring the perf result of parallel front end. Therefore, the measurement we choose is wall-time+non-relevent. (You may also need to exclude secondary Categories). In this case, the regression of #117435 should be 1.65%

pnkfelix · 2023-11-08T14:32:24Z

after followup in zulip for wg-parallel-rustc, they clarified that their expectation of a 1.65% regression was based on wall-time that includes so-called "non-relevant results". (Our default rustc-perf presentation filters out the non-relevant results, which will boost outliers and thus inflate the true mean effect.)
concluded that this was anticipated, and marking as triaged

@rustbot label: +perf-regression-triaged

SparrowLii · 2023-11-09T00:54:08Z

Thanks!

RalfJung · 2023-11-11T10:03:57Z

src/bootstrap/src/lib.rs

@@ -77,7 +77,7 @@ const LLD_FILE_NAMES: &[&str] = &["ld.lld", "ld64.lld", "lld-link", "wasm-ld"];
 ///
 /// If you make any major changes (such as adding new values or changing default values), please
 /// ensure that the associated PR ID is added to the end of this list.
-pub const CONFIG_CHANGE_HISTORY: &[usize] = &[115898, 116998];
+pub const CONFIG_CHANGE_HISTORY: &[usize] = &[115898, 116998, 117435];


When you do this, please explain in the PR description what we have to do to update our configs. I was sent here by bootstrap ("WARNING: there have been changes to x.py since you last updated") and now I have no idea what I'm supposed to do.

Update Rust toolchain from nightly-2023-11-06 to nightly-2023-11-07 without any other source changes. This is an automatically generated pull request. If any of the CI checks fail, manual intervention is required. In such a case, review the changes at https://github.com/rust-lang/rust from rust-lang@fee5518 up to rust-lang@189d6c7. The log for this commit range is: rust-lang@189d6c71f3 Auto merge of rust-lang#117641 - matthiaskrgr:rollup-f9c12td, r=matthiaskrgr rust-lang@9efe60b1eb Rollup merge of rust-lang#117637 - lqd:trivial-bounds-with-binder-vars, r=compiler-errors rust-lang@2a1f8bccee Rollup merge of rust-lang#117631 - smarnach:error-request-doc-fix, r=ChrisDenton rust-lang@8a34fea835 Rollup merge of rust-lang#117516 - matthiaskrgr:test_effects_113375_oob, r=fee1-dead rust-lang@1683df7d3b Rollup merge of rust-lang#117190 - matthiaskrgr:test_effects_113381, r=fee1-dead rust-lang@fb61292105 Auto merge of rust-lang#117292 - estebank:issue-80446, r=davidtwco rust-lang@4b7aacaa4f Silence redundant error on typo resulting on binop rust-lang@aea82b268a Auto merge of rust-lang#117603 - HKalbasi:make-feature-additive, r=Nilstrieb rust-lang@58351ae03f add test for trivial bound not holding in `soa-derive` rust-lang@2beca157c9 check binders with bound vars for global bounds that don't hold rust-lang@b049093560 Auto merge of rust-lang#116988 - RalfJung:null, r=WaffleLapkin rust-lang@e1fcecb1b9 Auto merge of rust-lang#117630 - matthiaskrgr:rollup-v0d5p3f, r=matthiaskrgr rust-lang@3a096e96fa Documentation cleanup for core::error::Request. rust-lang@b88d62e222 Rollup merge of rust-lang#117615 - bjorn3:misc_changes, r=davidtwco rust-lang@ec35c03143 Rollup merge of rust-lang#117613 - jackh726:vacation, r=compiler-errors rust-lang@60dca87cff Rollup merge of rust-lang#117592 - thomcc:env-span-wrong, r=davidtwco rust-lang@6bf2fb3679 Auto merge of rust-lang#117578 - compiler-errors:derive-encode-in-rustc_type_ir, r=davidtwco rust-lang@f9b644636f Auto merge of rust-lang#117435 - SparrowLii:nightly_parallel, r=oli-obk,davidtwco rust-lang@fcca978aa0 Auto merge of rust-lang#117607 - klensy:clang-17.0.4, r=Mark-Simulacrum rust-lang@152a4e90d1 Auto merge of rust-lang#117585 - dnbln:feat/move-kw-span, r=cjgillot rust-lang@f2a40e99ff use portable AtomicU64 for powerPC and MIPS rust-lang@7a892ab8d8 Auto merge of rust-lang#117576 - the8472:fix-io-copy-vec, r=Mark-Simulacrum rust-lang@c8a25eddfe Make the randomize feature of rustc_abi additive rust-lang@ec29a02071 Update doc comment for CodegenBackend::link rust-lang@ba82056a14 Use the actual computed crate name for -Zprint-vtable-sizes rust-lang@1a1b10fa63 Don't steal the parse query when using --pretty rust-lang@6e33e89337 Remove from vacation and compiler review group rust-lang@38ff91c453 bump clang version for dist-x86_64-linux from 17.0.2 to 17.0.4 rust-lang@86fca873ba Use the correct span when emitting the `env!` result rust-lang@c077147200 fix clippy author and failing test rust-lang@876f698790 Add the vis.visit_capture_by() in noop_visit_expr rust-lang@54ce0346c0 add `fn visit_capture_by` to MutVisitor and fix pprust-expr-roundtrip.rs rust-lang@df85b28b72 fixes for rustfmt + ast visitor rust-lang@a6b41aa6ba fmt rust-lang@241a654c07 Fix remaining uses of `CaptureBy::Value` rust-lang@8de489918b feat(hir): Store the `Span` of the `move` keyword rust-lang@bee7b5889e Derive TyEncodable/TyDecodable implementations that are parameterized over interner rust-lang@78aa5e511c detect EOF earlier rust-lang@8d8f06b277 avoid excessive initialization when copying to a Vec rust-lang@666cad9a5f add test for rust-lang#113375 rust-lang@384f7382ba add test for rust-lang#113381 rust-lang@248dd14fa5 update config.example.toml rust-lang@bf5fb7614b update bootstrap change history rust-lang@ab8101d019 enable parallel rustc in nightly builds rust-lang@231f93524d Detect misparsed binop caused by missing semi rust-lang@98d54da1ee document that the null pointer has the 0 address --------- Co-authored-by: celinval <celinval@users.noreply.github.com> Co-authored-by: Zyad Hassan <zyadh@amazon.com> Co-authored-by: Zyad Hassan <88045115+zhassan-aws@users.noreply.github.com>

enable parallel rustc in nightly builds

ab8101d

rustbot assigned oli-obk Oct 31, 2023

SparrowLii requested review from jackh726 and pietroalbini October 31, 2023 08:49

update bootstrap change history

bf5fb76

bjorn3 reviewed Oct 31, 2023

View reviewed changes

src/bootstrap/src/core/config/config.rs Show resolved Hide resolved

SparrowLii changed the title ~~enable parallel rustc in nightly builds~~ enable parallel rustc front end in nightly builds Oct 31, 2023

update config.example.toml

248dd14

oli-obk approved these changes Oct 31, 2023

View reviewed changes

SparrowLii mentioned this pull request Nov 1, 2023

Tracking Issue for Parallel Rustc Front-end #113349

Open

28 tasks

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 3, 2023

nnethercote mentioned this pull request Nov 3, 2023

Add parallel-rustc blog post. rust-lang/blog.rust-lang.org#1164

Merged

klensy reviewed Nov 3, 2023

View reviewed changes

src/ci/run.sh Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 3, 2023

rustbot assigned petrochenkov and unassigned oli-obk Nov 6, 2023

rustbot assigned davidtwco and unassigned petrochenkov Nov 6, 2023

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 6, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Nov 6, 2023

bors merged commit f9b6446 into rust-lang:master Nov 6, 2023
12 checks passed

rustbot added this to the 1.75.0 milestone Nov 6, 2023

This was referenced Nov 6, 2023

Weekly cargo update #112865

Merged

Add a new download-ci-llvm = if-unchanged option and enable it by default for profile = codegen #116881

Merged

LLVM Bitcode Linker: A self contained linker for nvptx and other targets #117458

Merged

rustbot added the perf-regression Performance regression. label Nov 6, 2023

github-actions bot mentioned this pull request Nov 7, 2023

Automated pull from upstream master ferrocene/ferrocene#89

Merged

rustbot added the perf-regression-triaged The performance regression has been triaged. label Nov 8, 2023

RalfJung reviewed Nov 11, 2023

View reviewed changes

Alexendoo mentioned this pull request Nov 24, 2023

Release nightly compilers with ability to internally parallelize #59667

Closed

bjorn3 mentioned this pull request Nov 24, 2023

[WIP] make nightly compilers able to parallelize #101566

Closed

SparrowLii mentioned this pull request Nov 25, 2023

refactor WorkerLocal for parallel compiler #109478

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable parallel rustc front end in nightly builds #117435

enable parallel rustc front end in nightly builds #117435

SparrowLii commented Oct 31, 2023 •

edited

Loading

rustbot commented Oct 31, 2023

rustbot commented Oct 31, 2023

oli-obk left a comment

lqd commented Nov 1, 2023 •

edited

Loading

oli-obk commented Nov 1, 2023

SparrowLii commented Nov 3, 2023 •

edited

Loading

bors commented Nov 3, 2023

matthiaskrgr commented Nov 3, 2023

bors commented Nov 3, 2023

This comment was marked as resolved.

davidtwco commented Nov 6, 2023

bors commented Nov 6, 2023

bors commented Nov 6, 2023

bors commented Nov 6, 2023

rust-timer commented Nov 6, 2023

pnkfelix commented Nov 7, 2023

SparrowLii commented Nov 8, 2023 •

edited

Loading

pnkfelix commented Nov 8, 2023

SparrowLii commented Nov 9, 2023

RalfJung Nov 11, 2023 •

edited

Loading

enable parallel rustc front end in nightly builds #117435

enable parallel rustc front end in nightly builds #117435

Conversation

SparrowLii commented Oct 31, 2023 • edited Loading

rustbot commented Oct 31, 2023

rustbot commented Oct 31, 2023

oli-obk left a comment

Choose a reason for hiding this comment

lqd commented Nov 1, 2023 • edited Loading

oli-obk commented Nov 1, 2023

SparrowLii commented Nov 3, 2023 • edited Loading

bors commented Nov 3, 2023

matthiaskrgr commented Nov 3, 2023

bors commented Nov 3, 2023

This comment was marked as resolved.

davidtwco commented Nov 6, 2023

bors commented Nov 6, 2023

bors commented Nov 6, 2023

bors commented Nov 6, 2023

rust-timer commented Nov 6, 2023

Overall result: ❌ regressions - ACTION NEEDED

Instruction count

Max RSS (memory usage)

Cycles

Binary size

pnkfelix commented Nov 7, 2023

SparrowLii commented Nov 8, 2023 • edited Loading

pnkfelix commented Nov 8, 2023

SparrowLii commented Nov 9, 2023

RalfJung Nov 11, 2023 • edited Loading

Choose a reason for hiding this comment

SparrowLii commented Oct 31, 2023 •

edited

Loading

lqd commented Nov 1, 2023 •

edited

Loading

SparrowLii commented Nov 3, 2023 •

edited

Loading

SparrowLii commented Nov 8, 2023 •

edited

Loading

RalfJung Nov 11, 2023 •

edited

Loading