Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests/run-make: failing tests get incorrectly ignored on rerun #129971

Closed
jieyouxu opened this issue Sep 5, 2024 · 0 comments · Fixed by #131681
Closed

tests/run-make: failing tests get incorrectly ignored on rerun #129971

jieyouxu opened this issue Sep 5, 2024 · 0 comments · Fixed by #131681
Labels
A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@jieyouxu
Copy link
Member

jieyouxu commented Sep 5, 2024 •
edited
Loading

First run (fails on native windows because obviously make not found):

running 362 tests
iii
[run-make] tests\run-make\cat-and-grep-sanity-check ... F
.iii.......ii..iii...........i...ii....i
[run-make] tests\run-make\emit-to-stdout ... F
....i........i..........ii...ii.ii.i.......  88/362
i.......ii..i...i....ii...i.....i..i....ii......ii.......i.i......i......i.....i......i. 176/362
..i....iii.iii...........i.i.iii..i....i.iii.i.......i..iiii.....i..................iii. 264/362
......ii.
[run-make] tests\run-make\split-debuginfo ... F
...i........i...........iii.ii.....ii.iiiiiiiiiiii..i..i...................... 352/362
..........

Second run (no changes):

running 362 tests
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii  88/362
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 176/362
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 264/362
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 352/362
iiiiiiiiii

test result: ok. 0 passed; 0 failed; 362 ignored; 0 measured; 0 filtered out; finished in 2.44ms

The same tests do fail with --force-rerun. Something about the ignore logic is incorrect.

We should not be ignoring failing tests.
@jieyouxu jieyouxu added A-testsuite Area: The testsuite used to check the correctness of rustc T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. C-bug Category: This is a bug. A-run-make Area: port run-make Makefiles to rmake.rs labels Sep 5, 2024
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Sep 5, 2024
@jieyouxu jieyouxu removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Sep 5, 2024
@jieyouxu jieyouxu mentioned this issue Oct 14, 2024
Merged
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Oct 14, 2024
@matthiaskrgr
Rollup merge of rust-lang#131681 - Zalathar:fix-run-make-stamp, r=jie…
55016c8 
…youxu

Fix up-to-date checking for run-make tests

This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.

Fixes rust-lang#129971.

r? jieyouxu
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Oct 14, 2024
@matthiaskrgr
Rollup merge of rust-lang#131681 - Zalathar:fix-run-make-stamp, r=jie…
ba74130 
…youxu

Fix up-to-date checking for run-make tests

This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.

Fixes rust-lang#129971.

r? jieyouxu
compiler-errors added a commit to compiler-errors/rust that referenced this issue Oct 15, 2024
@compiler-errors
Rollup merge of rust-lang#131681 - Zalathar:fix-run-make-stamp, r=jie…
71f2254 
…youxu

Fix up-to-date checking for run-make tests

This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.

Fixes rust-lang#129971.

r? jieyouxu
@bors bors closed this as completed in 258c177 Oct 15, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Oct 15, 2024
@rust-timer
Unrolled build for rust-lang#131681
5a95ee3 
Rollup merge of rust-lang#131681 - Zalathar:fix-run-make-stamp, r=jieyouxu

Fix up-to-date checking for run-make tests

This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.

Fixes rust-lang#129971.

r? jieyouxu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix up-to-date checking for run-make tests Zalathar/rust
2 participants
@jieyouxu @rustbot