extend the "if-unchanged" logic for compiler builds #131831

onur-ozkan · 2024-10-17T12:21:57Z

Implements the first item from this tracking issue.

In short, we want to make "if-unchanged" logic to check for changes outside of certain allowed directories, and this PR implements that.

See #131658 for more context.

rustbot · 2024-10-17T12:22:07Z

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2024-10-17T12:22:09Z

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

onur-ozkan · 2024-10-17T12:25:23Z

src/bootstrap/src/core/config/config.rs

+/// Holds subpaths from [`RUSTC_IF_UNCHANGED_ALLOWED_PATHS`] that should not be allowed.
+pub(crate) const RUSTC_IF_UNCHANGED_DENIED_SUBPATHS: &[&str] = &[


I don't want to have this list, but without it, it will be extremely hard to manage RUSTC_IF_UNCHANGED_ALLOWED_PATHS since there will be hundreds of paths 🙁.

I think it really doesn't matter if there are a few more rebuilds than necessary. I'd remove this list here and significantly reduce the list above. I think I'd be fine if it only contained library.

If we have to maintain a list of paths, making it simple and obvious and handle the most important cases seems way more important than making it granular and perfect.

bjorn3 · 2024-10-17T12:38:42Z

Would it make sense to compute the set of files that may not be changed for download-rustc to work based on the .d files in build/host/stage0-rustc? This is what cargo uses for determining if a crate should be recompiled. It doesn't list the Cargo.toml files though. For the aggregated .d file generated by cargo in build/host/stage0-rustc/<host>/release/rustc-main.d I think it would make sense to add Cargo.toml there too. These aggregated .d files are meant for integration with external build systems AFAIK, which need to track Cargo.toml changes too.

Noratrieb · 2024-10-17T12:41:14Z

src/bootstrap/src/core/config/config.rs

+    ":!CODE_OF_CONDUCT.md",
+    ":!CONTRIBUTING.md",
+    ":!COPYRIGHT",
+    ":!Cargo.lock",


this is wrong

if the dependencies of the compiler are updated, the compiler needs to be rebuilt

Oh, I didn't consider (but I should) the small bumps that can happen without changing Cargo.toml files in compiler and library tree.

yeah, there are so many things to consider or to not consider here that I think it makes the most sense to be super simple and only special case library as not rebuilding. in case anyone complains we can add more as needed

src/bootstrap/src/core/config/config.rs

onur-ozkan · 2024-10-18T04:39:22Z

@rustbot author

onur-ozkan · 2024-10-18T17:14:54Z

RUSTC_IF_UNCHANGED_DENIED_SUBPATHS is removed to rely on allowed paths only. Also, "src/bootstrap" is not included in the allowed paths on purpose since it can affect compiler builds.

@rustbot ready

onur-ozkan · 2024-10-18T17:21:50Z

Would it make sense to compute the set of files that may not be changed for download-rustc to work based on the .d files in build/host/stage0-rustc?

We are comparing against the nightly compiler, "stage0-rustc" wouldn't help.

bors · 2024-10-19T20:53:56Z

☔ The latest upstream changes (presumably #131934) made this pull request unmergeable. Please resolve the merge conflicts.

onur-ozkan · 2024-10-20T08:13:50Z

RUSTC_IF_UNCHANGED_DENIED_SUBPATHS is removed to rely on allowed paths only. Also, "src/bootstrap" is not included in the allowed paths on purpose since it can affect compiler builds.

@rustbot ready

Nevermind, allowed src/bootstrap again because modifications on bootstrap could break CI rustc usage and we wouldn't catch that if we used the in-tree compiler. The CI failure would only show up the next day.

src/bootstrap/src/core/config/config.rs

bors · 2024-10-23T23:21:04Z

☔ The latest upstream changes (presumably #132070) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-10-27T15:29:17Z

☔ The latest upstream changes (presumably #132215) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-10-31T01:35:02Z

☔ The latest upstream changes (presumably #132371) made this pull request unmergeable. Please resolve the merge conflicts.

onur-ozkan · 2024-11-10T20:58:06Z

FWIW, I ran my git analysis scripts using the current allowlist in this PR. For merge commits from the last 365 days, only around 3.8% would be cacheable. Before the latest change made a few minutes ago, it was 12.4%.

I guess src/tools is playing serious role on that.

What about PR CIs btw? I think CI rustc brings some serious advantage on that.

Signed-off-by: onur-ozkan <work@onurozkan.dev>

This reverts commit b3c2121.

onur-ozkan · 2024-11-11T08:19:45Z

Do you have any other concerns on this PR? @RalfJung

Signed-off-by: onur-ozkan <work@onurozkan.dev>

rustbot · 2024-11-11T08:28:18Z

This PR modifies config.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

…ozkan do not trust download-rustc=if-unchanged on CI for now See rust-lang#131658. Once rust-lang#131831 lands this will be unnecessary, for until then, better safe than sorry. r? `@onur-ozkan` Cc `@rust-lang/bootstrap`

RalfJung · 2024-11-12T09:29:49Z

I haven't dug into the logic here, so my concerns were entirely about the list of folders that considered "not affecting the compiler". That one has been resolved.

onur-ozkan · 2024-11-12T09:40:28Z

@bors r=Mark-Simulacrum

bors · 2024-11-12T09:40:31Z

📌 Commit 2d143ab has been approved by Mark-Simulacrum

It is now in the queue for this repository.

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#131831 (extend the "if-unchanged" logic for compiler builds) - rust-lang#132541 (Proper support for cross-crate recursive const stability checks) - rust-lang#132657 (AIX: add run-make support) - rust-lang#132901 (Warn about invalid `mir-enable-passes` pass names) - rust-lang#132923 (Triagebot: Consolidate the T-compiler ad hoc assignment groups) - rust-lang#132938 (Make precise capturing suggestion machine-applicable only if it has no APITs) - rust-lang#132947 (clarify `must_produce_diag` ICE for debugging) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#131831 - onur-ozkan:improve-rustc-if-unchanged-logic, r=Mark-Simulacrum extend the "if-unchanged" logic for compiler builds Implements the first item from [this tracking issue](rust-lang#131744). In short, we want to make "if-unchanged" logic to check for changes outside of certain allowed directories, and this PR implements that. See rust-lang#131658 for more context.

rustbot assigned Mark-Simulacrum Oct 17, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Oct 17, 2024

onur-ozkan commented Oct 17, 2024

View reviewed changes

Noratrieb reviewed Oct 17, 2024

View reviewed changes

src/bootstrap/src/core/config/config.rs Outdated Show resolved Hide resolved

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 18, 2024

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch from 2468461 to 135b903 Compare October 18, 2024 17:09

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 18, 2024

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch from 135b903 to 6e6625b Compare October 18, 2024 17:16

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch from 6e6625b to a140442 Compare October 20, 2024 01:08

Mark-Simulacrum reviewed Oct 20, 2024

View reviewed changes

src/bootstrap/src/core/config/config.rs Outdated Show resolved Hide resolved

src/bootstrap/src/core/config/config.rs Outdated Show resolved Hide resolved

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch 2 times, most recently from 31eaf3b to 7ccca59 Compare October 22, 2024 08:38

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch from 7ccca59 to b115071 Compare October 24, 2024 10:36

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch from b115071 to 7352257 Compare October 27, 2024 16:17

onur-ozkan removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 2, 2024

rustbot added the A-run-make Area: port run-make Makefiles to rmake.rs label Nov 10, 2024

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch from bbb7e28 to 8c01563 Compare November 10, 2024 20:54

This comment has been minimized.

Sign in to view

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch from 8c01563 to c104f3a Compare November 10, 2024 21:07

This comment has been minimized.

Sign in to view

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch 3 times, most recently from ef77e4d to f87fc33 Compare November 11, 2024 06:39

This comment has been minimized.

Sign in to view

onur-ozkan added 3 commits November 11, 2024 11:19

reduce RUSTC_IF_UNCHANGED_ALLOWED_PATHS significantly

6b38a15

Signed-off-by: onur-ozkan <work@onurozkan.dev>

move src/tools/build_helper into src/build_helper

bc75310

Signed-off-by: onur-ozkan <work@onurozkan.dev>

Revert "do not trust download-rustc=if-unchanged on CI for now"

4dfc052

This reverts commit b3c2121.

onur-ozkan force-pushed the improve-rustc-if-unchanged-logic branch from f87fc33 to 4dfc052 Compare November 11, 2024 08:19

update "if-unchanged" comments in config.example.toml

2d143ab

Signed-off-by: onur-ozkan <work@onurozkan.dev>

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 12, 2024

matthiaskrgr mentioned this pull request Nov 12, 2024

Rollup of 7 pull requests #132954

Merged

bors merged commit 134459e into rust-lang:master Nov 12, 2024
6 checks passed

rustbot added this to the 1.84.0 milestone Nov 12, 2024

onur-ozkan deleted the improve-rustc-if-unchanged-logic branch November 13, 2024 05:22

jieyouxu mentioned this pull request Nov 16, 2024

Use download-rustc=false global default, if-unchanged for tools and library profiles, and make rust.debug-assertions=true inhibit downloading CI rustc #133068

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

extend the "if-unchanged" logic for compiler builds #131831

extend the "if-unchanged" logic for compiler builds #131831

onur-ozkan commented Oct 17, 2024 •

edited

Loading

rustbot commented Oct 17, 2024

rustbot commented Oct 17, 2024

onur-ozkan Oct 17, 2024

Noratrieb Oct 17, 2024

Noratrieb Oct 17, 2024

bjorn3 commented Oct 17, 2024

Noratrieb Oct 17, 2024

onur-ozkan Oct 17, 2024

Noratrieb Oct 17, 2024

onur-ozkan Oct 17, 2024 •

edited

Loading

Noratrieb Oct 17, 2024

onur-ozkan commented Oct 18, 2024

onur-ozkan commented Oct 18, 2024

onur-ozkan commented Oct 18, 2024

bors commented Oct 19, 2024

onur-ozkan commented Oct 20, 2024

bors commented Oct 23, 2024

bors commented Oct 27, 2024

bors commented Oct 31, 2024

onur-ozkan commented Nov 10, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

onur-ozkan commented Nov 11, 2024

rustbot commented Nov 11, 2024

RalfJung commented Nov 12, 2024

onur-ozkan commented Nov 12, 2024

bors commented Nov 12, 2024

		/// Holds subpaths from [`RUSTC_IF_UNCHANGED_ALLOWED_PATHS`] that should not be allowed.
		pub(crate) const RUSTC_IF_UNCHANGED_DENIED_SUBPATHS: &[&str] = &[

extend the "if-unchanged" logic for compiler builds #131831

extend the "if-unchanged" logic for compiler builds #131831

Conversation

onur-ozkan commented Oct 17, 2024 • edited Loading

rustbot commented Oct 17, 2024

rustbot commented Oct 17, 2024

onur-ozkan Oct 17, 2024

Choose a reason for hiding this comment

Noratrieb Oct 17, 2024

Choose a reason for hiding this comment

Noratrieb Oct 17, 2024

Choose a reason for hiding this comment

bjorn3 commented Oct 17, 2024

Noratrieb Oct 17, 2024

Choose a reason for hiding this comment

onur-ozkan Oct 17, 2024

Choose a reason for hiding this comment

Noratrieb Oct 17, 2024

Choose a reason for hiding this comment

onur-ozkan Oct 17, 2024 • edited Loading

Choose a reason for hiding this comment

Noratrieb Oct 17, 2024

Choose a reason for hiding this comment

onur-ozkan commented Oct 18, 2024

onur-ozkan commented Oct 18, 2024

onur-ozkan commented Oct 18, 2024

bors commented Oct 19, 2024

onur-ozkan commented Oct 20, 2024

bors commented Oct 23, 2024

bors commented Oct 27, 2024

bors commented Oct 31, 2024

onur-ozkan commented Nov 10, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

onur-ozkan commented Nov 11, 2024

rustbot commented Nov 11, 2024

RalfJung commented Nov 12, 2024

onur-ozkan commented Nov 12, 2024

bors commented Nov 12, 2024

onur-ozkan commented Oct 17, 2024 •

edited

Loading

onur-ozkan Oct 17, 2024 •

edited

Loading