Skip to content

Fix breakage when running compiletest with --test-args=--edition=2015 #139578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 11, 2025

Conversation

pietroalbini
Copy link
Member

Compiletest has an --edition flag to change the default edition tests are run with. Unfortunately no test suite successfully executes when that flag is passed. If the edition is set to something greater than 2015 the breakage is expected, since the test suite currently supports only edition 2015 (Ferrous Systems will open an MCP about fixing that soonish). Surprisingly, the test suite is also broken if --edition=2015 is passed to compiletest. This PR focuses on fixing the latter.

This PR fixes the two categories of failures happening when --edition=2015 is passed:

  • Some edition-specific tests set their edition through //@ compile-flags instead of //@ edition. Compiletest doesn't parse the compile flags, so it would see no //@ edition and add another --edition flag, leading to a rustc error.
  • Compiletest would add the edition after //@ compile-flags, while some tests depend on flags passed to //@ compile-flags being the last flags in the rustc invocation.

Note that for the first category, I opted to manually go and replace all //@ compile-flags setting an edition with an explicit //@ edition. We could've changed compiletest to instead check whether an edition was set in //@ compile-flags, but I thought it was better to enforce a consistent way to set the edition in tests.

I also added the edition to the stamp, so that changing --edition results in tests being re-executed.

r? @jieyouxu

@rustbot rustbot added A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc PG-exploit-mitigations Project group: Exploit mitigations S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. T-rustdoc-frontend Relevant to the rustdoc-frontend team, which will review and decide on the web UI/UX output. WG-trait-system-refactor The Rustc Trait System Refactor Initiative (-Znext-solver) labels Apr 9, 2025
@rustbot
Copy link
Collaborator

rustbot commented Apr 9, 2025

Some changes occurred in src/tools/compiletest

cc @jieyouxu

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

Some changes occurred in tests/ui/sanitizer

cc @rcvalle

@rust-log-analyzer

This comment has been minimized.

@pietroalbini pietroalbini force-pushed the pa-compiletest-edition branch from aef870e to f13481b Compare April 9, 2025 14:50
Copy link
Member

@jieyouxu jieyouxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this mostly looks fine, just a question

@jieyouxu
Copy link
Member

@rustbot author

@rustbot rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 10, 2025
@rustbot
Copy link
Collaborator

rustbot commented Apr 10, 2025

Reminder, once the PR becomes ready for a review, use @rustbot ready.

@pietroalbini pietroalbini force-pushed the pa-compiletest-edition branch from f13481b to 9cf6e8c Compare April 10, 2025 08:02
@pietroalbini
Copy link
Member Author

@rustbot author

@pietroalbini
Copy link
Member Author

...why did I do that...

@rustbot review

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 10, 2025
Copy link
Member

@jieyouxu jieyouxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, you can r=me after PR CI is green.

bors added a commit to rust-lang-ci/rust that referenced this pull request Apr 10, 2025
…jieyouxu

Fix breakage when running compiletest with `--test-args=--edition=2015`

Compiletest has an `--edition` flag to change the default edition tests are run with. Unfortunately no test suite successfully executes when that flag is passed. If the edition is set to something greater than 2015 the breakage is expected, since the test suite currently supports only edition 2015 (Ferrous Systems will open an MCP about fixing that soonish). Surprisingly, the test suite is also broken if `--edition=2015` is passed to compiletest. This PR focuses on fixing the latter.

This PR fixes the two categories of failures happening when `--edition=2015` is passed:

* Some edition-specific tests set their edition through `//@ compile-flags` instead of `//@ edition`. Compiletest doesn't parse the compile flags, so it would see no `//@ edition` and add another `--edition` flag, leading to a rustc error.
* Compiletest would add the edition after `//@ compile-flags`, while some tests depend on flags passed to `//@ compile-flags` being the last flags in the rustc invocation.

Note that for the first category, I opted to manually go and replace all `//@ compile-flags` setting an edition with an explicit `//@ edition`. We could've changed compiletest to instead check whether an edition was set in `//@ compile-flags`, but I thought it was better to enforce a consistent way to set the edition in tests.

I also added the edition to the stamp, so that changing `--edition` results in tests being re-executed.

r? `@jieyouxu`
@bors
Copy link
Collaborator

bors commented Apr 10, 2025

⌛ Testing commit 33794fd with merge c8e41ba...

@rust-log-analyzer

This comment has been minimized.

@bors
Copy link
Collaborator

bors commented Apr 10, 2025

💔 Test failed - checks-actions

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 10, 2025
@pietroalbini
Copy link
Member Author

@bors r=jieyouxu

@bors
Copy link
Collaborator

bors commented Apr 11, 2025

📌 Commit 3ebf1c2 has been approved by jieyouxu

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 11, 2025
@bors
Copy link
Collaborator

bors commented Apr 11, 2025

⌛ Testing commit 3ebf1c2 with merge 71b68da...

@bors
Copy link
Collaborator

bors commented Apr 11, 2025

☀️ Test successful - checks-actions
Approved by: jieyouxu
Pushing 71b68da to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 11, 2025
@bors bors merged commit 71b68da into rust-lang:master Apr 11, 2025
7 checks passed
@rustbot rustbot added this to the 1.88.0 milestone Apr 11, 2025
Copy link

What is this? This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 81d8c74 (parent) -> 71b68da (this PR)

Test differences

No test diffs found

Job duration changes

  1. x86_64-apple-1: 10842.6s -> 7302.7s (-32.6%)
  2. dist-x86_64-apple: 11315.8s -> 9156.7s (-19.1%)
  3. aarch64-apple: 3956.5s -> 3622.7s (-8.4%)
  4. x86_64-apple-2: 4197.1s -> 3911.8s (-6.8%)
  5. dist-x86_64-netbsd: 5206.0s -> 4950.8s (-4.9%)
  6. dist-various-1: 4571.1s -> 4353.3s (-4.8%)
  7. dist-apple-various: 6916.2s -> 6598.4s (-4.6%)
  8. dist-i686-msvc: 7194.6s -> 6915.9s (-3.9%)
  9. i686-msvc-2: 7067.3s -> 7337.6s (3.8%)
  10. aarch64-gnu: 6619.9s -> 6866.5s (3.7%)
How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

@rust-timer
Copy link
Collaborator

Finished benchmarking commit (71b68da): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary 2.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
2.2% [2.2%, 2.2%] 1
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) 2.2% [2.2%, 2.2%] 1

Cycles

Results (secondary 4.6%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
4.6% [3.8%, 5.3%] 2
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) - - 0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 778.797s -> 778.939s (0.02%)
Artifact size: 365.95 MiB -> 365.97 MiB (0.00%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-compiletest Area: The compiletest test runner A-rustc-dev-guide Area: rustc-dev-guide A-testsuite Area: The testsuite used to check the correctness of rustc merged-by-bors This PR was explicitly merged by bors. PG-exploit-mitigations Project group: Exploit mitigations S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. T-rustdoc-frontend Relevant to the rustdoc-frontend team, which will review and decide on the web UI/UX output. WG-trait-system-refactor The Rustc Trait System Refactor Initiative (-Znext-solver)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants