avoid compiler_for for dist tools and force the current compiler

[onur-ozkan]

Using compiler_for in dist steps was causing to install stage1 tools into the dist tarballs, which doesn't match with the stage2 compiler.

Fixes #137469

[rustbot]

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[Noratrieb]

makes sense that this needs to be changed and that this used to be wrong, since if you put the tools in the wrong sysroot when building, you also have to get them out of this wrong sysroot again :D

[Noratrieb]

(@rust-lang/bootstrap if someone has time to review this today that would be nice and avoid the need for a revert)

[jieyouxu]

I'll take a look.

[jieyouxu]

Thanks, the changes are obviously correct w.r.t. dist tools in the dist flow.

But, from what I can tell, this may partially regress --force-use-stage={1,2} in which case it may result in more builds than the user explicitly instructed, but fixing the dist flow is way more important.

To be safe, what I'll do is to run a try-job on a dist job and then inspect the dist toolchain via rustup-toolchain-install-master (since we have some time before the next nightly).

[jieyouxu]

@bors try (for a dist toolchain produced under the CI environment)

[bors]

⌛ Trying commit 4a1f7a1 with merge 39ca7bb...

[jieyouxu]

Please ping me once the try build is complete (if I don't look at that soon).

[onur-ozkan]

There is one error I got on my local environment:

Dist rustc-nightly-x86_64-unknown-linux-gnu
        finished in 20.822 seconds
ERROR: Unable to find the stamp file /home/nimda/devspace/.other/rustc-builds/build/x86_64-unknown-linux-gnu/stage2-std/x86_64-unknown-linux-gnu/release/.libstd-stamp, did you try to keep
 a nonexistent build stage?

looking on the fix.

[onur-ozkan]

Everything should work as expected now.

[jieyouxu]

@onur-ozkan need to <>-ize the link it seems

error: this URL is not a hyperlink
    --> src/bootstrap/src/core/builder/mod.rs:1266:65
     |
1266 |     /// FIXME: This function is unnecessary (and dangerous, see https://github.com/rust-lang/rust/issues/137469).
     |                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     |
     = note: bare URLs are not automatically turned into clickable links
     = note: `-D rustdoc::bare-urls` implied by `-D warnings`
     = help: to override `-D warnings` add `#[allow(rustdoc::bare_urls)]`
help: use an automatic link instead
     |
1266 |     /// FIXME: This function is unnecessary (and dangerous, see <https://github.com/rust-lang/rust/issues/137469>).
     |   

[onur-ozkan]

Trying again with the current changes @bors try

[jieyouxu]

I tried ./x dist with a dist profile locally, and I can reproduce the dist clippy/rustc looking for the same librustc_driver. So that part LGTM. Waiting for the try dist toolchain to come back so we can double-check. If that comes back also as expected, then please r=me.

[onur-ozkan]

@bors r=jieyouxu

[bors]

📌 Commit 1c7b60f has been approved by jieyouxu

It is now in the queue for this repository.

[bors]

⌛ Testing commit 1c7b60f with merge f8a913b...

[matthiaskrgr]

the job was still running, and probably just 10-20 mins from completion, you can see the latest progress here
https://github.com/rust-lang-ci/rust/actions/runs/13485121545/job/37675331794#step:27:38379

for some jobs github only displays the first couple thousand lines but does not update the live-log anymore unfortunately.

The longest usually take around 3 hours - 3hours 30 mins until they are done.

[tgross35]

Ah, sorry about that then. Is there any way to know whether the job is actually stuck vs. just appearing stuck? It had already exceeded its usual run time by half an hour, which is unusual in any case.

[bors]

☀️ Test successful - checks-actions
Approved by: jieyouxu
Pushing f8a913b to master...

[rust-timer]

Finished benchmarking commit (f8a913b): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results (primary 4.4%, secondary -1.9%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	4.4%	[3.2%, 5.7%]	2
Regressions ❌ (secondary)	2.1%	[2.0%, 2.2%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.5%	[-2.8%, -2.2%]	12
All ❌✅ (primary)	4.4%	[3.2%, 5.7%]	2

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 770.364s -> 770.257s (-0.01%)
Artifact size: 359.79 MiB -> 359.67 MiB (-0.03%)

[onur-ozkan]

It made it just in time. f8a913b is the merge commit for this PR :)

[Noratrieb]

thank you for the fix!

[ehuss]

To be clear, do we all understand that this PR has significantly increased CI time?

For example, powerpc64le-unknown-linux-gnu is now the longest running job at over 3h.

A significant difference is that it builds stage2 host (x86_64-unknown-linux-gnu) rustc (and other expensive things like wasm-component-ld). That adds about 15 minutes. That seems unnecessary, since it should be fine to use stage1 rustc to build the other two hosts.

[onur-ozkan]

The previous logic was somewhat magical and unpredictable. If we want powerpc64le-unknown-linux-gnu to use stage 1 we should explicitly configure that runner to build and use stage 1.

[ehuss]

Is that possible after this PR?

./x dist --stage=2 --host=... --target=... does the wrong thing:

• Compiles build compiler from stage0 to stage1 (ok)
• Compiles host compiler from stage1 to stage2(host) (ok)
• Compiles host compiler from stage0 to stage1(host) (wrong, why is it doing this?)
• Compiles build compiler from stage1 to stage2 (wrong, why is it doing this?)

./x dist --stage=1 --host=... --target=... also does the wrong thing:

• Compiles host compiler from stage0 to stage1(host) (wrong, should be from stage1)
• Compiles build compiler from stage0 to stage1 (ok)
• Compiles host compiler from stage1 to stage2(host) (ok)

My expectations are:

• Cross compiling versus not cross compiling should not require different stage settings. But you seem to be saying that we need to use different commands for those?
• When compiling (either cross or not), it should:
  • build stage1 build compiler from stage0(build)
  • build stage2 host compiler from stage1(build)

From what I can tell, it seems to work correctly when not cross compiling.

[onur-ozkan]

Ah, I got it wrong.

Seems like we need to add some additional logic to handle cross-compilations. I will look on that during the week.

[jieyouxu]

I missed that during review. I think what's happening here is that compiler_for has additional magic that adjusts which rustc we need to ensure (in particular, it has specific logic to adjust rustc's staging accounting for target I think). I'll also take a look.

[jieyouxu]

I opened #138004 to track this, because I keep having trouble finding this PR discussion 😓

[onur-ozkan]

Fixing this easy but I need to make sure it doesn't regress #137469 on cross target dist tools/components.

I will dive into this tomorrow.

[onur-ozkan]

PR is up: #138039