Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

allow bypassing the build directory lock #118069

Merged
merged 1 commit into from
Dec 10, 2023
Merged

allow bypassing the build directory lock #118069

merged 1 commit into from
Dec 10, 2023

Conversation

onur-ozkan
Copy link
Member

@onur-ozkan onur-ozkan commented Nov 19, 2023 •
edited
Loading

As bootstrap locks its entire build directory, parallel bootstrapping for anything becomes impossible. This change enables developers to bypass the locking mechanism (with --bypass-bootstrap-lock flag) when it is unnecessary for their specific use case.

more context: https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Build.20.28miri.3F.29.20sysroots.20in.20parallel

cc @saethlin

@rustbot
Copy link
Collaborator

rustbot commented Nov 19, 2023

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Nov 19, 2023
@rustbot
Copy link
Collaborator

rustbot commented Nov 19, 2023

This PR modifies src/bootstrap/src/core/config. If appropriate, please also update CONFIG_CHANGE_HISTORY in src/bootstrap/src/lib.rs and change-id in config.example.toml.

@onur-ozkan onur-ozkan force-pushed the bypass_bootstrap_lock branch 2 times, most recently from cc66292 to ca37e43 Compare November 19, 2023 19:33
Mark-Simulacrum
Mark-Simulacrum approved these changes Dec 9, 2023
View reviewed changes
Copy link
Member

@Mark-Simulacrum Mark-Simulacrum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r=me with nit fixed

src/bootstrap/src/core/config/flags.rs Outdated
@@ -133,6 +133,13 @@ pub struct Flags {
/// whether to use color in cargo and rustc output
pub color: Color,

#[arg(global(true), long)]
/// Bootstrap uses this value to decide whether it should lock or bypass locking its build directory.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess "build directory" here feels misleading. We are using this to protect the entire build -- right? In particular, it is not safe to pass this option just because you've already built rustbuild itself.

I'd feel better removing the "directory" phrasing here and in 1-2 other places.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, "its build directory" feels like directory for the bootstrap artifacts rather than compiler/std/tool ones.

@onur-ozkan
allow bypassing the build directory lock
6860654 
As bootstrap locks its entire build directory, parallel bootstrapping
for anything becomes impossible. This change enables developers to bypass
the locking mechanism when it is unnecessary for their specific use case.

Signed-off-by: onur-ozkan <work@onurozkan.dev>
@onur-ozkan
Copy link
Member Author

@bors r=Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Dec 9, 2023

📌 Commit 6860654 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 9, 2023
@bors
Copy link
Contributor

bors commented Dec 9, 2023

⌛ Testing commit 6860654 with merge 8cd8d31...

@bors
Copy link
Contributor

bors commented Dec 10, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 8cd8d31 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Dec 10, 2023
@bors bors merged commit 8cd8d31 into rust-lang:master Dec 10, 2023
12 checks passed
@rustbot rustbot added this to the 1.76.0 milestone Dec 10, 2023
This was referenced Dec 10, 2023
Merged
Closed
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (8cd8d31): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 1.9% [0.6%, 2.8%] 4
Regressions ❌
(secondary)		 - - 0
Improvements ✅
(primary)		 -0.7% [-0.9%, -0.5%] 2
Improvements ✅
(secondary)		 -2.1% [-2.1%, -2.1%] 1
All ❌✅ (primary) 1.0% [-0.9%, 2.8%] 6

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 - - 0
Improvements ✅
(primary)		 - - 0
Improvements ✅
(secondary)		 -1.8% [-1.8%, -1.8%] 1
All ❌✅ (primary) - - 0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 668.865s -> 668.296s (-0.09%)
Artifact size: 314.11 MiB -> 314.15 MiB (0.01%)

celinval added a commit to celinval/rust-dev that referenced this pull request Jun 4, 2024
@github-actions @celinval
Automatic toolchain upgrade to nightly-2023-12-11 (rust-lang#2932)
813d012 
Update Rust toolchain from nightly-2023-12-10 to nightly-2023-12-11
without any other source changes.
This is an automatically generated pull request. If any of the CI checks
fail, manual intervention is required. In such a case, review the
changes at https://github.com/rust-lang/rust from
rust-lang@06e02d5
up to
rust-lang@d86d65b.
The log for this commit range is:
rust-lang@d86d65bbc1 Auto merge of
rust-lang#118368 - GuillaumeGomez:env-flag, r=Nilstrieb
rust-lang@ec4176167b Auto merge of
rust-lang#118703 - Kobzol:bootstrap-config-unused, r=onur-ozkan
rust-lang@f1c5558edc Add ChangeInfo
record
rust-lang@ccbd88dc83 Remove unused
run_dsymutil and gpg_password_file config values
rust-lang@6badc0d840 Destructure TOML
configs
rust-lang@7e452c123c Auto merge of
rust-lang#118791 - saethlin:use-immediate-type, r=nikic
rust-lang@b9068315db Auto merge of
rust-lang#116952 - compiler-errors:lifetime_capture_rules_2024, r=TaKO8Ki
rust-lang@befd1eb4ec Auto merge of
rust-lang#116278 - Kobzol:bootstrap-lld-mode, r=albertlarsan68,petrochenkov
rust-lang@dc2f77aad6 Add (unstable)
documentation for `--env` command line option
rust-lang@d2b1f94f05 Add feature gate
test for `--env` flag
rust-lang@37d68093da Add ui tests for
`--env` option
rust-lang@486e55e547 Implement `--env`
compiler flag
rust-lang@53031b264e Review fixes
rust-lang@84f6130fe3 Auto merge of
rust-lang#118692 - surechen:remove_unused_imports, r=petrochenkov
rust-lang@4750e9de47 Produce an explicit
error when using a self-contained lld, but we don't add it to sysroot
rust-lang@ea769dbeb7 Add change tracker
entry
rust-lang@cbfe6327a1 Do not invoke
external lld to figure out thread flags in self-contained mode
rust-lang@50865745e1 Update
`config.example.toml`
rust-lang@40c3d351ad Use MCP510
rust-lang@48c1607bc6 Introduce `LldMode`
and generalize parsing of `use-lld`
rust-lang@d9f9e67bc1 Refactor rust(do)c
linker flags
rust-lang@b3c9ffdc77 Add `is_msvc`
function
rust-lang@c1a3919378 Auto merge of
rust-lang#118792 - naglis:fix-mutex-doc-typo, r=workingjubilee
rust-lang@42dfac5e08 Auto merge of
rust-lang#118788 - compiler-errors:const-pretty, r=fee1-dead
rust-lang@61afc9c928 Auto merge of
rust-lang#116949 - hamza1311:stablize-arc_unwrap_or_clone, r=dtolnay
rust-lang@c71c246876 Auto merge of
rust-lang#118550 - cjgillot:filecheck-const-prop, r=Mark-Simulacrum
rust-lang@40ae34194c remove redundant
imports
rust-lang@f7253f2317 Auto merge of
rust-lang#118787 - GuillaumeGomez:rollup-fj5wr3q, r=GuillaumeGomez
rust-lang@7d50a39763 Fix typo in
`std::sync::Mutex` example
rust-lang@8cd8d31369 Auto merge of
rust-lang#118069 - onur-ozkan:bypass_bootstrap_lock, r=Mark-Simulacrum
rust-lang@b0a580112b Use
immediate_backend_type when reading from a const alloc
rust-lang@afa35e90ef Print constness in
TraitPredPrintModifiersAndPath
rust-lang@7467c3a45c
s/const_effect/host_effect
rust-lang@f1bf874fb1 Don't print host
effect param in pretty path_generic_args
rust-lang@6860654d82 allow bypassing the
build directory lock
rust-lang@dd234696ed Rollup merge of
rust-lang#118782 - jyn514:powershell, r=ChrisDenton
rust-lang@034d73d6d7 Rollup merge of
rust-lang#118775 - Young-Flash:fix, r=compiler-errors
rust-lang@5b9e917b64 Rollup merge of
rust-lang#118774 - lcnr:region-refactor-uwu, r=compiler-errors
rust-lang@cc821d3ae6 Rollup merge of
rust-lang#118747 - Urgau:check-cfg-freebsd-cleanup, r=onur-ozkan
rust-lang@83e814f88c Rollup merge of
rust-lang#117966 - lxy19980601:safe_compilation_options, r=Mark-Simulacrum
rust-lang@2cf54e9f99 use `&` instead of
start-process in x.ps1
rust-lang@ef796db5f0 add test for
inductive cycle hangs
rust-lang@cb6984217f chore: add test case
for type with generic
rust-lang@0f40b6545d Remove extra
check-cfg handled by libc directly
rust-lang@803772e81d Enable new capture
rules by default on edition 2024
rust-lang@acba7efe1b Add test for
implicitly capturing late-bound var with new capture rules
rust-lang@0ad160a585 Add
lifetime_capture_rules_2024
rust-lang@3f8487a099 Add safe compilation
options
rust-lang@30a95b7c0a FileCheck
while_let_loops.
rust-lang@c00068e49f FileCheck
tuple_literal_propagation.
rust-lang@87522d0007 FileCheck
return_place.
rust-lang@a12027e128 FileCheck
switch_int.
rust-lang@19767eb7a6 FileCheck slice_len.
rust-lang@3e90c1b434 FileCheck
scalar_literal_propagation.
rust-lang@f3743aec51 FileCheck repeat.
rust-lang@343ef6a9cb FileCheck
reify_fn_ptr.
rust-lang@6baec3ccc2 FileCheck ref_deref.
rust-lang@c8c9207e4c FileCheck
read_immutable_static.
rust-lang@45dd5d6bf3 FileCheck
mutable_variable_unprop_assign.
rust-lang@6a8eea8f5b FileCheck
mutable_variable_aggregate_partial_read.
rust-lang@d91bb5074e FileCheck
mutable_variable_no_prop.
rust-lang@3e169abc1b FileCheck
mutable_variable_aggregate_mut_ref.
rust-lang@03c5ad1549 FileCheck
mutable_variable_aggregate.
rust-lang@ea9f968333 FileCheck
mutable_variable.
rust-lang@902a3e2e75 FileCheck
mult_by_zero.
rust-lang@8e9b912c4c FileCheck
issue_67019.
rust-lang@ce9b1e23a5 FileCheck
issue_66971.
rust-lang@218d8ccf43 FileCheck
inherit_overflow.
rust-lang@6086dd6766 FileCheck indirect.
rust-lang@bf5d114da8 FileCheck
discriminant.
rust-lang@043d29b58a FileCheck and rename
const_prop_fails_gracefully.
rust-lang@7f328d2a44 FileCheck
checked_add.
rust-lang@e6a1b77cd1 FileCheck cast.
rust-lang@b8f2f63931 FileCheck boxes.
rust-lang@3fc03948a8 FileCheck
boolean_identities.
rust-lang@e8e35c8127 FileCheck
bad_op_unsafe_oob_for_slices.
rust-lang@97f03cb898 FileCheck
bad_op_mod_by_zero.
rust-lang@0d5bc872a9 FileCheck
bad_op_div_by_zero.
rust-lang@9f01d9d1b6 FileCheck
array_index.
rust-lang@6564bac532 FileCheck aggregate.
rust-lang@378abbc604 FileCheck
address_of_pair.
rust-lang@540921e468 Stablize
arc_unwrap_or_clone

Co-authored-by: celinval <celinval@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Mark-Simulacrum Mark-Simulacrum Mark-Simulacrum approved these changes

Assignees

@Mark-Simulacrum Mark-Simulacrum

Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Milestone
1.76.0
Development

Successfully merging this pull request may close these issues.
5 participants
@onur-ozkan @rustbot @bors @rust-timer @Mark-Simulacrum