Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

-Zpanic_abort_tests fails with benchmarks as test #73509

Closed
Talanor opened this issue Jun 19, 2020 · 2 comments · Fixed by #111992
Closed

-Zpanic_abort_tests fails with benchmarks as test #73509

Talanor opened this issue Jun 19, 2020 · 2 comments · Fixed by #111992
Assignees
@pietroalbini
Labels
A-libtest Area: `#[test]` / the `test` library C-bug Category: This is a bug. requires-nightly This issue requires a nightly compiler in some way.

Comments

@Talanor
Copy link

Talanor commented Jun 19, 2020 •
edited
Loading

Hi, I have a minimalistic test case where cargo test panics:

#![feature(test)]

#[cfg(test)]
mod tests {
    extern crate test;

    #[bench]
    fn noname(_b: &mut test::Bencher) {
    }
}

with a new project initialized with cargo init --lib and

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
export RUSTDOCFLAGS="-Cpanic=abort"

as specified in the grcov readme
The error I get is :

$ cargo clean && cargo build && cargo test
   Compiling grcovtest v0.1.0 (XXXX/grcovtest)
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
   Compiling grcovtest v0.1.0 (XXXX/grcovtest)
    Finished test [unoptimized + debuginfo] target(s) in 0.19s
     Running target/debug/deps/grcovtest-4d053526558b4910

running 1 test
thread 'main' panicked at 'Cannot run dynamic test fn out-of-process', src/libtest/lib.rs:496:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: test failed, to rerun pass '--lib'

Caused by:
  process didn't exit successfully: `XXXX/grcovtest/target/debug/deps/grcovtest-4d053526558b4910` (signal: 6, SIGABRT: process abort signal)

and with --lib:

$ cargo clean && cargo build && cargo test --lib     
   Compiling grcovtest v0.1.0 (XXXX/grcovtest)
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
   Compiling grcovtest v0.1.0 (XXXX/grcovtest)
    Finished test [unoptimized + debuginfo] target(s) in 0.22s
     Running target/debug/deps/grcovtest-4d053526558b4910

running 1 test
thread 'main' panicked at 'Cannot run dynamic test fn out-of-process', src/libtest/lib.rs:496:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: test failed, to rerun pass '--lib'

Caused by:
  process didn't exit successfully: `XXXX/grcovtest/target/debug/deps/grcovtest-4d053526558b4910` (signal: 6, SIGABRT: process abort signal)

cargo test runs fine if I remove the -Cpanic=abort from the rustflags

My understanding is that it goes in https://github.com/rust-lang/rust/blob/master/src/libtest/lib.rs#L496 instead of https://github.com/rust-lang/rust/blob/master/src/libtest/lib.rs#L485 but I have no idea why or how to circumvent it.

Thank you.

Meta

$ rustc --version --verbose                                                                                              
rustc 1.46.0-nightly (feb3536eb 2020-06-09)
binary: rustc
commit-hash: feb3536eba10c2e4585d066629598f03d5ddc7c6
commit-date: 2020-06-09
host: x86_64-unknown-linux-gnu
release: 1.46.0-nightly
LLVM version: 10.0
Backtrace (with RUST_BACKTRACE=full) 

$ cargo clean && cargo build && cargo test --lib
   Compiling grcovtest v0.1.0 (XXXX/grcovtest)
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
   Compiling grcovtest v0.1.0 (XXXX/grcovtest)
    Finished test [unoptimized + debuginfo] target(s) in 0.21s
     Running target/debug/deps/grcovtest-4d053526558b4910

running 1 test
thread 'main' panicked at 'Cannot run dynamic test fn out-of-process', src/libtest/lib.rs:496:22
stack backtrace:
   0:     0x557e9e6e00c5 - backtrace::backtrace::libunwind::trace::hc3f32ba24d88242a
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1:     0x557e9e6e00c5 - backtrace::backtrace::trace_unsynchronized::h1f075597100358e7
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2:     0x557e9e6e00c5 - std::sys_common::backtrace::_print_fmt::h4397782fec59f199
                               at src/libstd/sys_common/backtrace.rs:78
   3:     0x557e9e6e00c5 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h39652f0242246c81
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x557e9e71c7fc - core::fmt::write::h540ac4a6a1232abc
                               at src/libcore/fmt/mod.rs:1076
   5:     0x557e9e6d2843 - std::io::Write::write_fmt::haa03b5883ee9e297
                               at src/libstd/io/mod.rs:1537
   6:     0x557e9e6e50e0 - std::sys_common::backtrace::_print::ha6d9bdf9f438acec
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x557e9e6e50e0 - std::sys_common::backtrace::print::ha1fba17a8e5f6bc7
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x557e9e6e50e0 - std::panicking::default_hook::{{closure}}::hbf9fcecd42c17d90
                               at src/libstd/panicking.rs:198
   9:     0x557e9e6e4e2c - std::panicking::default_hook::h39b8550780094336
                               at src/libstd/panicking.rs:218
  10:     0x557e9e6e5767 - std::panicking::rust_panic_with_hook::hc8b48ce48004c103
                               at src/libstd/panicking.rs:477
  11:     0x557e9e677a6e - std::panicking::begin_panic::h87c3ab8d799b6998
                               at /rustc/feb3536eba10c2e4585d066629598f03d5ddc7c6/src/libstd/panicking.rs:404
  12:     0x557e9e69ca87 - test::run_test::ha53930a547443f35
                               at src/libtest/lib.rs:496
  13:     0x557e9e687b2c - test::run_tests::h30f17a5394720b03
                               at src/libtest/lib.rs:299
  14:     0x557e9e687b2c - test::console::run_tests_console::h0afa09c552374ad5
                               at src/libtest/console.rs:280
  15:     0x557e9e697469 - test::test_main::hf995bbd176ccef4a
                               at src/libtest/lib.rs:120
  16:     0x557e9e698adb - test::test_main_static_abort::h9e5c26ff93351d49
                               at src/libtest/lib.rs:171
  17:     0x557e9e670466 - grcovtest::main::ha17a8198f2d29754
  18:     0x557e9e66f493 - std::rt::lang_start::{{closure}}::h2dfa0fe676b19ed6
                               at XXXX/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  19:     0x557e9e6e5bf8 - std::rt::lang_start_internal::{{closure}}::hcf986077e2da8d6b
                               at src/libstd/rt.rs:52
  20:     0x557e9e6e5bf8 - std::panicking::try::do_call::h076edd10f2a91969
                               at src/libstd/panicking.rs:297
  21:     0x557e9e6e5bf8 - std::panicking::try::h54343dd1a9d830aa
                               at src/libstd/panicking.rs:274
  22:     0x557e9e6e5bf8 - std::panic::catch_unwind::h81799eb8afacafdd
                               at src/libstd/panic.rs:394
  23:     0x557e9e6e5bf8 - std::rt::lang_start_internal::hf4abd869024c3270
                               at src/libstd/rt.rs:51
  24:     0x557e9e66f40e - std::rt::lang_start::h37ccdd487e4b5e45
                               at XXXX/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  25:     0x557e9e67049a - main
  26:     0x7f03850f1002 - __libc_start_main
  27:     0x557e9e66f2ce - _start
  28:                0x0 - <unknown>
error: test failed, to rerun pass '--lib'

Caused by:
  process didn't exit successfully: `XXXX/grcovtest/target/debug/deps/grcovtest-4d053526558b4910` (signal: 6, SIGABRT: process abort signal)
@Talanor Talanor added the C-bug Category: This is a bug. label Jun 19, 2020
@jonas-schievink jonas-schievink added A-libtest Area: `#[test]` / the `test` library requires-nightly This issue requires a nightly compiler in some way. labels Jun 19, 2020
@ehuss ehuss changed the title cargo test panics with -Cpanic=abort on bench -Zpanic_abort_tests fails with benchmarks as test Jun 21, 2020
@ehuss ehuss mentioned this issue Jun 21, 2020
Open
@Twister915
Copy link

I also ran into this issue

@ehuss ehuss mentioned this issue Oct 19, 2022
Closed
@tmandry tmandry mentioned this issue Oct 20, 2022
Merged
@pietroalbini pietroalbini self-assigned this May 25, 2023
@pietroalbini pietroalbini mentioned this issue May 26, 2023
Merged
@pietroalbini
Copy link
Member

Opened #111992 to fix this issue!

TaKO8Ki added a commit to TaKO8Ki/rust that referenced this issue Jun 27, 2023
@TaKO8Ki
Rollup merge of rust-lang#111992 - ferrocene:pa-panic-abort-tests-ben…
1d7f087 
…ch, r=m-ou-se

Test benchmarks with `-Z panic-abort-tests`

During test execution, when a `#[bench]` benchmark is encountered it's executed once to check whether it works. Unfortunately that was not compatible with `-Z panic-abort-tests`: the feature works by spawning a subprocess for each test, which prevents the use of dynamic tests as we cannot pass closures to child processes, and before this PR the conversion from benchmark to test was done by turning benchmarks into dynamic tests whose closures execute the benchmark once.

The approach this PR took was to add two new kinds of `TestFn`s: `StaticBenchAsTestFn` and `DynBenchAsTestFn` (:warning: **this is a breaking change** :warning:). With that change, a `StaticBenchFn` can be converted into a `StaticBenchAsTestFn` without creating dynamic tests, and making it possible to test `#[bench]` functions with `-Z panic-abort-tests`. The subprocess test runner also had to be updated to perform the conversion from benchmark to test when appropriate.

Along with the bug fix, in the first commit I refactored how tests are executed: rather than executing the test function in multiple places across `libtest`, there is now a private `TestFn::into_runnable()` method, which returns either a `RunnableTest` or `RunnableBench`, on which you can call the `run()` method. This simplified the rest of the changes in the PR.

This PR is best reviewed commit-by-commit.
Fixes rust-lang#73509
@bors bors closed this as completed in e5bb341 Jul 1, 2023
github-actions bot pushed a commit to rust-lang/miri that referenced this issue Jul 2, 2023
@bors
Auto merge of #111992 - ferrocene:pa-panic-abort-tests-bench, r=m-ou-se
e941a7c 
Test benchmarks with `-Z panic-abort-tests`

During test execution, when a `#[bench]` benchmark is encountered it's executed once to check whether it works. Unfortunately that was not compatible with `-Z panic-abort-tests`: the feature works by spawning a subprocess for each test, which prevents the use of dynamic tests as we cannot pass closures to child processes, and before this PR the conversion from benchmark to test was done by turning benchmarks into dynamic tests whose closures execute the benchmark once.

The approach this PR took was to add two new kinds of `TestFn`s: `StaticBenchAsTestFn` and `DynBenchAsTestFn` (:warning: **this is a breaking change** :warning:). With that change, a `StaticBenchFn` can be converted into a `StaticBenchAsTestFn` without creating dynamic tests, and making it possible to test `#[bench]` functions with `-Z panic-abort-tests`. The subprocess test runner also had to be updated to perform the conversion from benchmark to test when appropriate.

Along with the bug fix, in the first commit I refactored how tests are executed: rather than executing the test function in multiple places across `libtest`, there is now a private `TestFn::into_runnable()` method, which returns either a `RunnableTest` or `RunnableBench`, on which you can call the `run()` method. This simplified the rest of the changes in the PR.

This PR is best reviewed commit-by-commit.
Fixes rust-lang/rust#73509
thomcc pushed a commit to tcdi/postgrestd that referenced this issue Aug 24, 2023
@bors
Auto merge of #111992 - ferrocene:pa-panic-abort-tests-bench, r=m-ou-se
cebb579 
Test benchmarks with `-Z panic-abort-tests`

During test execution, when a `#[bench]` benchmark is encountered it's executed once to check whether it works. Unfortunately that was not compatible with `-Z panic-abort-tests`: the feature works by spawning a subprocess for each test, which prevents the use of dynamic tests as we cannot pass closures to child processes, and before this PR the conversion from benchmark to test was done by turning benchmarks into dynamic tests whose closures execute the benchmark once.

The approach this PR took was to add two new kinds of `TestFn`s: `StaticBenchAsTestFn` and `DynBenchAsTestFn` (:warning: **this is a breaking change** :warning:). With that change, a `StaticBenchFn` can be converted into a `StaticBenchAsTestFn` without creating dynamic tests, and making it possible to test `#[bench]` functions with `-Z panic-abort-tests`. The subprocess test runner also had to be updated to perform the conversion from benchmark to test when appropriate.

Along with the bug fix, in the first commit I refactored how tests are executed: rather than executing the test function in multiple places across `libtest`, there is now a private `TestFn::into_runnable()` method, which returns either a `RunnableTest` or `RunnableBench`, on which you can call the `run()` method. This simplified the rest of the changes in the PR.

This PR is best reviewed commit-by-commit.
Fixes rust-lang/rust#73509
lnicola pushed a commit to lnicola/rust-analyzer that referenced this issue Apr 7, 2024
@bors
Auto merge of #111992 - ferrocene:pa-panic-abort-tests-bench, r=m-ou-se
c8c90d9 
Test benchmarks with `-Z panic-abort-tests`

During test execution, when a `#[bench]` benchmark is encountered it's executed once to check whether it works. Unfortunately that was not compatible with `-Z panic-abort-tests`: the feature works by spawning a subprocess for each test, which prevents the use of dynamic tests as we cannot pass closures to child processes, and before this PR the conversion from benchmark to test was done by turning benchmarks into dynamic tests whose closures execute the benchmark once.

The approach this PR took was to add two new kinds of `TestFn`s: `StaticBenchAsTestFn` and `DynBenchAsTestFn` (:warning: **this is a breaking change** :warning:). With that change, a `StaticBenchFn` can be converted into a `StaticBenchAsTestFn` without creating dynamic tests, and making it possible to test `#[bench]` functions with `-Z panic-abort-tests`. The subprocess test runner also had to be updated to perform the conversion from benchmark to test when appropriate.

Along with the bug fix, in the first commit I refactored how tests are executed: rather than executing the test function in multiple places across `libtest`, there is now a private `TestFn::into_runnable()` method, which returns either a `RunnableTest` or `RunnableBench`, on which you can call the `run()` method. This simplified the rest of the changes in the PR.

This PR is best reviewed commit-by-commit.
Fixes rust-lang/rust#73509
RalfJung pushed a commit to RalfJung/rust-analyzer that referenced this issue Apr 27, 2024
@bors
Auto merge of #111992 - ferrocene:pa-panic-abort-tests-bench, r=m-ou-se
f697b36 
Test benchmarks with `-Z panic-abort-tests`

During test execution, when a `#[bench]` benchmark is encountered it's executed once to check whether it works. Unfortunately that was not compatible with `-Z panic-abort-tests`: the feature works by spawning a subprocess for each test, which prevents the use of dynamic tests as we cannot pass closures to child processes, and before this PR the conversion from benchmark to test was done by turning benchmarks into dynamic tests whose closures execute the benchmark once.

The approach this PR took was to add two new kinds of `TestFn`s: `StaticBenchAsTestFn` and `DynBenchAsTestFn` (:warning: **this is a breaking change** :warning:). With that change, a `StaticBenchFn` can be converted into a `StaticBenchAsTestFn` without creating dynamic tests, and making it possible to test `#[bench]` functions with `-Z panic-abort-tests`. The subprocess test runner also had to be updated to perform the conversion from benchmark to test when appropriate.

Along with the bug fix, in the first commit I refactored how tests are executed: rather than executing the test function in multiple places across `libtest`, there is now a private `TestFn::into_runnable()` method, which returns either a `RunnableTest` or `RunnableBench`, on which you can call the `run()` method. This simplified the rest of the changes in the PR.

This PR is best reviewed commit-by-commit.
Fixes rust-lang/rust#73509
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pietroalbini pietroalbini

Labels
A-libtest Area: `#[test]` / the `test` library C-bug Category: This is a bug. requires-nightly This issue requires a nightly compiler in some way.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Test benchmarks with -Z panic-abort-tests ferrocene/rust
4 participants
@jonas-schievink @Twister915 @Talanor @pietroalbini