Implement RFC3137 trim-paths sysroot changes - take 2 #129687

Urgau · 2024-08-28T14:28:19Z

This PR is a continuation of #118149. Nothing really changed, except for #129408 which I was able to trigger locally.

Original description:

Implement parts of #111540

Right now, backtraces into sysroot always shows /rustc/$hash in diagnostics, e.g.
thread 'main' panicked at 'hello world', map-panic.rs:2:50
stack backtrace:
   0: std::panicking::begin_panic
             at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:616:12
   1: map_panic::main::{{closure}}
             at ./map-panic.rs:2:50
   2: core::option::Option<T>::map
             at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:929:29
   3: map_panic::main
             at ./map-panic.rs:2:30
   4: core::ops::function::FnOnce::call_once
             at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
RFC 3127 said

We want to change this behaviour such that, when rust-src source files can be discovered, the virtual path is discarded and therefore the local path will be embedded, unless there is a --remap-path-prefix that causes this local path to be remapped in the usual way.

This PR implements this behaviour. When rust-src is present at compile time, rustc replaces /rustc/$hash with a real path into local rust-src with best effort. To sanitise this, users must explicitly supply --remap-path-prefix=<path to rust-src>=foo.

cc @cbeuw
Fix #105907
Fix #85463

try-job: dist-x86_64-linux
try-job: x86_64-msvc
try-job: dist-x86_64-msvc
try-job: armhf-gnu

Urgau · 2024-08-28T15:47:10Z

@bors try

Implement RFC3137 trim-paths sysroot changes - take 2 Continuation of rust-lang#118149 *Draft for the time being, to figure out if the previous issues have been fixed.* r? ghost try-job: dist-x86_64-linux try-job: x86_64-msvc try-job: armhf-gnu

bors · 2024-08-28T15:48:22Z

⌛ Trying commit cac4a30 with merge a600fe8...

bors · 2024-08-28T17:40:38Z

☀️ Try build successful - checks-actions
Build commit: a600fe8 (a600fe8b275a62c2973e9fc1b589e50302eb2540)

Urgau · 2024-08-29T09:54:53Z

Great, the extended try is successful! dist-x86_64-linux and x86_64-msvc were the builder that failed in the previous PR: #118149 (comment).

I've verified locally (with rustup-toolchain-install-master):

that the prebuild rlibs only contains /rustc/ paths ✔
that compiling tests/codegen/remap_path_prefix/issue-73167-remap-std.rs without the rust-src component only leads to /rustc/ paths ✔
that compiling tests/codegen/remap_path_prefix/issue-73167-remap-std.rs with the rust-src component leads to many (not all) paths redirected to /home/gh-Urgau/.rustup/toolchains/a600fe8b275a62c2973e9fc1b589e50302eb2540/lib/rustlib/src/rust/library/ ✔

Everything looks go.
r? compiler

rustbot · 2024-08-29T09:54:57Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

jieyouxu

Thanks, the changes themselves look good to me. We definitely should update rustc-dev-guide to mention existence and purpose of {{rust-src-base}}, would be very cool if you can open a follow-up dev-guide PR.

jieyouxu · 2024-09-25T13:11:53Z

compiler/rustc_metadata/src/rmeta/decoder.rs

+                    const STD_LIBS: &[&str] = &[
+                        "core",
+                        "alloc",
+                        "std",
+                        "test",
+                        "term",
+                        "unwind",
+                        "proc_macro",
+                        "panic_abort",
+                        "panic_unwind",
+                        "profiler_builtins",
+                        "rtstartup",
+                        "rustc-std-workspace-core",
+                        "rustc-std-workspace-alloc",
+                        "rustc-std-workspace-std",
+                        "backtrace",
+                    ];


Remark: I think this is fine, though it might be interesting if stdlib ever adds/removes/renames specific crates. I also can't immediately think of a good way to add a test to make sure this list is always in sync.

jieyouxu · 2024-09-25T13:16:53Z

src/tools/compiletest/src/header.rs

@@ -1115,6 +1115,7 @@ fn expand_variables(mut value: String, config: &Config) -> String {
    const CWD: &str = "{{cwd}}";
    const SRC_BASE: &str = "{{src-base}}";
    const BUILD_BASE: &str = "{{build-base}}";
+    const RUST_SRC_BASE: &str = "{{rust-src-base}}";


Suggestion: we definitely want to update rustc-dev-guide's compiletest directive remarks to mention {{rust-src-base}}, which is not to be confused with {{src-base}}.

Opened rust-lang/rustc-dev-guide#2079

jieyouxu · 2024-09-25T13:21:58Z

@bors r+ rollup

bors · 2024-09-25T13:22:03Z

📌 Commit cac4a30 has been approved by jieyouxu

It is now in the queue for this repository.

Implement RFC3137 trim-paths sysroot changes - take 2 This PR is a continuation of rust-lang#118149. Nothing really changed, except for rust-lang#129408 which I was able to trigger locally. Original description: > Implement parts of rust-lang#111540 > > Right now, backtraces into sysroot always shows /rustc/$hash in diagnostics, e.g. > > ``` > thread 'main' panicked at 'hello world', map-panic.rs:2:50 > stack backtrace: > 0: std::panicking::begin_panic > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:616:12 > 1: map_panic::main::{{closure}} > at ./map-panic.rs:2:50 > 2: core::option::Option<T>::map > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:929:29 > 3: map_panic::main > at ./map-panic.rs:2:30 > 4: core::ops::function::FnOnce::call_once > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5 > note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. > ``` > > [RFC 3127 said](https://rust-lang.github.io/rfcs/3127-trim-paths.html#changing-handling-of-sysroot-path-in-rustc) > > > We want to change this behaviour such that, when rust-src source files can be discovered, the virtual path is discarded and therefore the local path will be embedded, unless there is a --remap-path-prefix that causes this local path to be remapped in the usual way. > > This PR implements this behaviour. When `rust-src` is present at compile time, rustc replaces /rustc/$hash with a real path into local rust-src with best effort. To sanitise this, users must explicitly supply `--remap-path-prefix=<path to rust-src>=foo`. cc `@cbeuw` Fix rust-lang#105907 Fix rust-lang#85463 try-job: dist-x86_64-linux try-job: x86_64-msvc try-job: armhf-gnu

…iaskrgr Rollup of 5 pull requests Successful merges: - rust-lang#129687 (Implement RFC3137 trim-paths sysroot changes - take 2) - rust-lang#129759 (Stabilize `const_refs_to_static`) - rust-lang#130329 (Reorder stack spills so that constants come later.) - rust-lang#130845 (Utf8Chunks: add link to Utf8Chunk) - rust-lang#130846 (Revert Break into the debugger on panic (129019)) r? `@ghost` `@rustbot` modify labels: rollup

Implement RFC3137 trim-paths sysroot changes - take 2 This PR is a continuation of rust-lang#118149. Nothing really changed, except for rust-lang#129408 which I was able to trigger locally. Original description: > Implement parts of rust-lang#111540 > > Right now, backtraces into sysroot always shows /rustc/$hash in diagnostics, e.g. > > ``` > thread 'main' panicked at 'hello world', map-panic.rs:2:50 > stack backtrace: > 0: std::panicking::begin_panic > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:616:12 > 1: map_panic::main::{{closure}} > at ./map-panic.rs:2:50 > 2: core::option::Option<T>::map > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:929:29 > 3: map_panic::main > at ./map-panic.rs:2:30 > 4: core::ops::function::FnOnce::call_once > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5 > note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. > ``` > > [RFC 3127 said](https://rust-lang.github.io/rfcs/3127-trim-paths.html#changing-handling-of-sysroot-path-in-rustc) > > > We want to change this behaviour such that, when rust-src source files can be discovered, the virtual path is discarded and therefore the local path will be embedded, unless there is a --remap-path-prefix that causes this local path to be remapped in the usual way. > > This PR implements this behaviour. When `rust-src` is present at compile time, rustc replaces /rustc/$hash with a real path into local rust-src with best effort. To sanitise this, users must explicitly supply `--remap-path-prefix=<path to rust-src>=foo`. cc ``@cbeuw`` Fix rust-lang#105907 Fix rust-lang#85463 try-job: dist-x86_64-linux try-job: x86_64-msvc try-job: armhf-gnu

…iaskrgr Rollup of 6 pull requests Successful merges: - rust-lang#129687 (Implement RFC3137 trim-paths sysroot changes - take 2) - rust-lang#130820 (Fix diagnostics for coroutines with () as input.) - rust-lang#130833 (Fix the misleading diagnostic for `let_underscore_drop` on type without `Drop` implementation) - rust-lang#130845 (Utf8Chunks: add link to Utf8Chunk) - rust-lang#130868 (Update FIXME comment in s390x_unknown_linux_*.rs) - rust-lang#130873 (rustc_target: Add powerpc64 atomic-related features) r? `@ghost` `@rustbot` modify labels: rollup

Implement RFC3137 trim-paths sysroot changes - take 2 This PR is a continuation of rust-lang#118149. Nothing really changed, except for rust-lang#129408 which I was able to trigger locally. Original description: > Implement parts of rust-lang#111540 > > Right now, backtraces into sysroot always shows /rustc/$hash in diagnostics, e.g. > > ``` > thread 'main' panicked at 'hello world', map-panic.rs:2:50 > stack backtrace: > 0: std::panicking::begin_panic > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:616:12 > 1: map_panic::main::{{closure}} > at ./map-panic.rs:2:50 > 2: core::option::Option<T>::map > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:929:29 > 3: map_panic::main > at ./map-panic.rs:2:30 > 4: core::ops::function::FnOnce::call_once > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5 > note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. > ``` > > [RFC 3127 said](https://rust-lang.github.io/rfcs/3127-trim-paths.html#changing-handling-of-sysroot-path-in-rustc) > > > We want to change this behaviour such that, when rust-src source files can be discovered, the virtual path is discarded and therefore the local path will be embedded, unless there is a --remap-path-prefix that causes this local path to be remapped in the usual way. > > This PR implements this behaviour. When `rust-src` is present at compile time, rustc replaces /rustc/$hash with a real path into local rust-src with best effort. To sanitise this, users must explicitly supply `--remap-path-prefix=<path to rust-src>=foo`. cc ```@cbeuw``` Fix rust-lang#105907 Fix rust-lang#85463 try-job: dist-x86_64-linux try-job: x86_64-msvc try-job: armhf-gnu

bors · 2024-09-27T15:21:14Z

⌛ Testing commit 61ef983 with merge cde36fb...

Implement RFC3137 trim-paths sysroot changes - take 2 This PR is a continuation of rust-lang#118149. Nothing really changed, except for rust-lang#129408 which I was able to trigger locally. Original description: > Implement parts of rust-lang#111540 > > Right now, backtraces into sysroot always shows /rustc/$hash in diagnostics, e.g. > > ``` > thread 'main' panicked at 'hello world', map-panic.rs:2:50 > stack backtrace: > 0: std::panicking::begin_panic > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:616:12 > 1: map_panic::main::{{closure}} > at ./map-panic.rs:2:50 > 2: core::option::Option<T>::map > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:929:29 > 3: map_panic::main > at ./map-panic.rs:2:30 > 4: core::ops::function::FnOnce::call_once > at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5 > note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. > ``` > > [RFC 3127 said](https://rust-lang.github.io/rfcs/3127-trim-paths.html#changing-handling-of-sysroot-path-in-rustc) > > > We want to change this behaviour such that, when rust-src source files can be discovered, the virtual path is discarded and therefore the local path will be embedded, unless there is a --remap-path-prefix that causes this local path to be remapped in the usual way. > > This PR implements this behaviour. When `rust-src` is present at compile time, rustc replaces /rustc/$hash with a real path into local rust-src with best effort. To sanitise this, users must explicitly supply `--remap-path-prefix=<path to rust-src>=foo`. cc `@cbeuw` Fix rust-lang#105907 Fix rust-lang#85463 try-job: dist-x86_64-linux try-job: x86_64-msvc try-job: dist-x86_64-msvc try-job: armhf-gnu

bors · 2024-09-27T17:27:40Z

💔 Test failed - checks-actions

jieyouxu · 2024-09-28T11:45:57Z

I'm not exactly sure why opt-dist sysroot path prefixes becomes wonky...

I suspect we may need to modify opt-dist to pass some information to bootstrap -> compiletest to possibly repair the path prefix up to something like unpacked-dist/rustc-nightly-x86_64-unknown-linux-gnu/rustc/lib/rustlib/src/rust, but I haven't dug deeply into this yet.

We wouldn't want to skip these tests naively especially if it can possibly reveal bugs in PGO+LTO among other things.

in case the real paths into the libstd/libcore are located inside the the build directory, maybe because it's coming from an extracted dist component in the build dir (cc opt-dist)

This is done to cover the path of the test it-self as it may not live on the same root directory as {{rust-src-base}}, which can be the case if {{rust-src-base}} is coming from a extracted dist build (cc opt-dist)

jieyouxu · 2024-09-28T21:19:59Z

If the opt-dist tests steps never get reached in PR CI or try-jobs, let's rollup=never and send it off for full CI. r=me once you drop the CI commit.

@bors rollup=never (expected to fail)

Urgau · 2024-09-28T21:26:45Z

@bors r=@jieyouxu

bors · 2024-09-28T21:26:47Z

📌 Commit 3a6c6ee has been approved by jieyouxu

It is now in the queue for this repository.

bors · 2024-09-29T03:04:08Z

⌛ Testing commit 3a6c6ee with merge 1d9162b...

bors · 2024-09-29T05:53:38Z

☀️ Test successful - checks-actions
Approved by: jieyouxu
Pushing 1d9162b to master...

rust-timer · 2024-09-29T08:31:55Z

Finished benchmarking commit (1d9162b): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.2%, 0.2%]	1
Regressions ❌ (secondary)	0.7%	[0.2%, 1.2%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[0.2%, 0.2%]	1

Max RSS (memory usage)

Results (primary 2.8%, secondary -0.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.8%	[2.8%, 2.8%]	1
Regressions ❌ (secondary)	3.5%	[2.2%, 6.6%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.4%	[-4.0%, -1.4%]	6
All ❌✅ (primary)	2.8%	[2.8%, 2.8%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

Results (primary 0.2%, secondary 0.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.0%, 1.4%]	71
Regressions ❌ (secondary)	0.1%	[0.0%, 0.2%]	18
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[0.0%, 1.4%]	71

Bootstrap: 768.272s -> 768.678s (0.05%)
Artifact size: 341.40 MiB -> 341.38 MiB (-0.01%)

Urgau marked this pull request as ready for review August 29, 2024 09:54

rustbot assigned fmease Aug 29, 2024

Urgau added the F-trim-paths Feature: trim-paths label Aug 29, 2024

jieyouxu assigned jieyouxu and unassigned fmease Sep 20, 2024

jieyouxu approved these changes Sep 25, 2024

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 25, 2024

This comment was marked as resolved.

Sign in to view

Urgau mentioned this pull request Sep 25, 2024

Add documentation for {{rust-src-base}} rust-lang/rustc-dev-guide#2079

Merged

matthiaskrgr mentioned this pull request Sep 25, 2024

Rollup of 5 pull requests #130854

Closed

matthiaskrgr mentioned this pull request Sep 26, 2024

Rollup of 6 pull requests #130878

Closed

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 27, 2024

This comment has been minimized.

Sign in to view

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 27, 2024

Urgau added 2 commits September 28, 2024 21:44

compiletest: normalize to $SRC_DIR_REAL before $TEST_BUILD_DIR

37ea27c

in case the real paths into the libstd/libcore are located inside the the build directory, maybe because it's coming from an extracted dist component in the build dir (cc opt-dist)

remap-path-prefix-sysroot: remap {{src-base}} as well

3a6c6ee

This is done to cover the path of the test it-self as it may not live on the same root directory as {{rust-src-base}}, which can be the case if {{rust-src-base}} is coming from a extracted dist build (cc opt-dist)

rustbot added the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label Sep 28, 2024

This comment has been minimized.

Sign in to view

Urgau force-pushed the rfc3127-sysroot-2 branch from 9c89706 to 3a6c6ee Compare September 28, 2024 21:25

Urgau removed the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label Sep 28, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 28, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Sep 29, 2024

bors merged commit 1d9162b into rust-lang:master Sep 29, 2024
7 checks passed

rustbot added this to the 1.83.0 milestone Sep 29, 2024

Urgau added relnotes Marks issues that should be documented in the release notes of the next release. and removed T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Sep 29, 2024

rustbot mentioned this pull request Sep 29, 2024

Tracking issue for release notes of #129687: Implement RFC3137 trim-paths sysroot changes - take 2 #131015

Open

3 tasks

jieyouxu added the CI-spurious-fail-msvc CI spurious failure: target env msvc label Oct 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement RFC3137 trim-paths sysroot changes - take 2 #129687

Implement RFC3137 trim-paths sysroot changes - take 2 #129687

Urgau commented Aug 28, 2024 •

edited by tgross35

Loading

Urgau commented Aug 28, 2024

bors commented Aug 28, 2024

bors commented Aug 28, 2024

Urgau commented Aug 29, 2024

rustbot commented Aug 29, 2024

jieyouxu left a comment •

edited

Loading

jieyouxu Sep 25, 2024

jieyouxu Sep 25, 2024

Urgau Sep 25, 2024

jieyouxu commented Sep 25, 2024

bors commented Sep 25, 2024

This comment was marked as resolved.

bors commented Sep 27, 2024

This comment has been minimized.

bors commented Sep 27, 2024

jieyouxu commented Sep 28, 2024

This comment has been minimized.

jieyouxu commented Sep 28, 2024

Urgau commented Sep 28, 2024

bors commented Sep 28, 2024

bors commented Sep 29, 2024

bors commented Sep 29, 2024

rust-timer commented Sep 29, 2024

Implement RFC3137 trim-paths sysroot changes - take 2 #129687

Implement RFC3137 trim-paths sysroot changes - take 2 #129687

Conversation

Urgau commented Aug 28, 2024 • edited by tgross35 Loading

Urgau commented Aug 28, 2024

bors commented Aug 28, 2024

bors commented Aug 28, 2024

Urgau commented Aug 29, 2024

rustbot commented Aug 29, 2024

jieyouxu left a comment • edited Loading

Choose a reason for hiding this comment

jieyouxu Sep 25, 2024

Choose a reason for hiding this comment

jieyouxu Sep 25, 2024

Choose a reason for hiding this comment

Urgau Sep 25, 2024

Choose a reason for hiding this comment

jieyouxu commented Sep 25, 2024

bors commented Sep 25, 2024

This comment was marked as resolved.

bors commented Sep 27, 2024

This comment has been minimized.

bors commented Sep 27, 2024

jieyouxu commented Sep 28, 2024

This comment has been minimized.

jieyouxu commented Sep 28, 2024

Urgau commented Sep 28, 2024

bors commented Sep 28, 2024

bors commented Sep 29, 2024

bors commented Sep 29, 2024

rust-timer commented Sep 29, 2024

Overall result: ❌ regressions - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Urgau commented Aug 28, 2024 •

edited by tgross35

Loading

jieyouxu left a comment •

edited

Loading