Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic (attempt to subtract with overflow) when calculating the number of passed_tests in utils\render_tests.rs #136607

Closed
anforowicz opened this issue Feb 5, 2025 · 7 comments · Fixed by #136630
Assignees
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-libs Relevant to the library team, which will review and decide on the PR/issue.

Comments

@anforowicz
Copy link
Contributor

I apologize for not following the bug template + for not having a consistent repro. I hope that opening this issue will still help (writing down and sharing what we know, looping in the author of the line that panics, etc.).

It seems that when running rust tests on Windows, the test binary sometimes (flakily, inconsistently) panics saying (based on two cases when it happened recently - see here and here:

 �[1m�[32m     Running�[0m tests\builtin-clone.rs (build\x86_64-pc-windows-msvc\stage1-std\x86_64-pc-windows-msvc\release\deps\builtin_clone-92ab8a9329f9b62b.exe)
 
 running 1 tests
 
 thread 'main' panicked at src\bootstrap\src\utils\render_tests.rs:230:36:
 attempt to subtract with overflow
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

So far this has only happened on Windows (and not on Linux). And it seems to only be happening when running tests from library/std/tests/builtin-clone.rs (which contain only a single testcase).

It seems that the line that panics has been introduced relatively recently (around a month ago) in dc11857#diff-218273cbe7a566fb9d2f12187307eb7fd0e4664abd7c375f0f4cd30183ea78c9R230:

```
let passed_tests = self.executed_tests - (self.failures.len() + self.ignored_tests);
```

I guess the panic indicates that self.executed_tests is smaller than the sum of self.failures.len() and self.ignored_tests. This seems rather unexpected.

FWIW This issue is tracked in Chromium's issue tracker as https://crbug.com/392978169.

@anforowicz anforowicz added the C-bug Category: This is a bug. label Feb 5, 2025
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Feb 5, 2025
@anforowicz
Copy link
Contributor Author

/cc @ranger-ross from #135355

@ehuss
Copy link
Contributor

ehuss commented Feb 6, 2025

It looks like there are multiple issues here.

One is that the stdout is getting corrupted because multiple threads are writing to it at once. You can see this here:

 C:\b\s\w\ir\cache\builder\src\third_party\rust-src\library\std>{ "type": "test", "name": "sync::mpsc::tests::oneshot_single_thread_try_send_closed", "event": "ok" }

bootstrap misses that JSON message because it got mixed with some other output. I don't know what is printing all that Microsoft Windows [Version 10.0.19045.5131] stuff.

The other issue is that Renderer is not resetting all its counters when it switches between suites. Currently it sets a few fields here like self.executed_tests. However, it does not clear self.ignored_tests or self.failures.

Because it missed one of the messages, it did not reach 100% in the percentage. When it runs the builtin-clone test, the executed_tests is 1, but ignored_tests is something like 5.

@ehuss
Copy link
Contributor

ehuss commented Feb 6, 2025

The tests that are writing to the output are test_creation_flags and test_proc_thread_attributes. They are spawning cmd without capturing the output. I think those tests should be fixed to capture the output so it doesn't interfere with the JSON output.

There's still a bunch of other tests that are printing panic messages from spawned threads (like test_join_panic). I don't remember if the panic printing code locks stdout. If it does, then it is unlikely to interfere with the JSON output. If not, then it has some risk of having the same problem. (It also just makes the test output kinda noisy, but it might not be easy to avoid.)

@saethlin saethlin added the A-testsuite Area: The testsuite used to check the correctness of rustc label Feb 6, 2025
@jieyouxu jieyouxu self-assigned this Feb 6, 2025
@jieyouxu jieyouxu added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-libs Relevant to the library team, which will review and decide on the PR/issue. and removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Feb 6, 2025
@jieyouxu

This comment has been minimized.

@bors bors closed this as completed in 5b22425 Feb 6, 2025
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Feb 6, 2025
Rollup merge of rust-lang#136630 - jieyouxu:render_tests, r=ChrisDenton

Change two std process tests to not output to std{out,err}, and fix test suite stat reset in bootstrap CI test rendering

I don't really know how to test if this unbreaks CI (since rust-lang#136607 reported that this breaks the CI test rendering *sometimes*).

Fixes rust-lang#136607.

r? `@ChrisDenton` (two Windows process tests, but feel free to reroll)
@ehuss
Copy link
Contributor

ehuss commented Feb 6, 2025

I'm going to reopen because there are two issues here. The other is that the counters never get reset between test suites, which means they can show incorrect results and possibly break in the same way depending on how the tests are organized.

@ehuss ehuss reopened this Feb 6, 2025
@jieyouxu
Copy link
Member

jieyouxu commented Feb 6, 2025

The other is that the counters never get reset between test suites, which means they can show incorrect results and possibly break in the same way depending on how the tests are organized.

@ehuss AFAIK they should be reset if we receive a new test suite start message now:

fn render_message(&mut self, message: Message) {
match message {
Message::Suite(SuiteMessage::Started { test_count }) => {
println!("\nrunning {test_count} tests");
self.benches = vec![];
self.failures = vec![];
self.ignored_tests = 0;
self.executed_tests = 0;
self.terse_tests_in_line = 0;
self.tests_count = Some(test_count);
}

Did I miss anything? Or is there another example I can use to test?

@ehuss
Copy link
Contributor

ehuss commented Feb 6, 2025

Oh, I'm sorry! I did not look at the PR, and I think the title got truncated on my thing and I just assumed it was just updating the tests. My bad!

@ehuss ehuss closed this as completed Feb 6, 2025
github-actions bot pushed a commit to tautschnig/verify-rust-std that referenced this issue Mar 11, 2025
Change two std process tests to not output to std{out,err}, and fix test suite stat reset in bootstrap CI test rendering

I don't really know how to test if this unbreaks CI (since rust-lang#136607 reported that this breaks the CI test rendering *sometimes*).

Fixes rust-lang#136607.

r? `@ChrisDenton` (two Windows process tests, but feel free to reroll)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-libs Relevant to the library team, which will review and decide on the PR/issue.
Projects
None yet
5 participants