Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

more usable minimizer output? #38

Open
Tracked by #60
Gankra opened this issue Jun 30, 2024 · 1 comment
Open
Tracked by #60

more usable minimizer output? #38

Gankra opened this issue Jun 30, 2024 · 1 comment
Labels
enhancement New feature or request help wanted Extra attention is needed question Further information is requested

Comments

@Gankra
Copy link
Owner

Gankra commented Jun 30, 2024

When a test fails in the check step (so the test compiled and ran, but we didn't like the result we got out), we do a second pass, minimizing all the failures:

abi-cafe/src/main.rs

Lines 242 to 253 in 6ba6865

fn generate_minimized_failures(
cfg: &Config,
harness: &Arc<TestHarness>,
rt: &tokio::runtime::Runtime,
reports: &FullReport,
) {
info!("rerunning failures");
let tasks = reports.tests.iter().flat_map(|report| {
let Some(check) = report.results.check.as_ref() else {
return vec![];
};
check

As a side-effect of building them we'll print out their paths, but we don't produce any summary:

abi-cafe/src/main.rs

Lines 306 to 309 in 6ba6865

let _results = tasks
.into_iter()
.map(|task| rt.block_on(task).expect("failed to join task"))
.collect::<Vec<_>>();

We also, more importantly to me, don't actually produce a fully usable reproduction, we only produce two source files, and it's your problem to make use of them. In an ideal world we would generate a cargo and/or (C?)Make project you could build and run.

To that end at one point I tossed a "main.rs" in the harness includes, but I never sorted out the details. Honestly not sure what would be useful to people reporting/receiving bug reports.

//! This is the primary file for the abi-cafe standalone binary/project mode.
//!
//! This mode is primarily intended for reporting/debugging abi-cafe test failures,
//! where you want the particulars of abi-cafe to go away, and want a minimized
//! reproduction of the issue.
//!
//! As such this is incompatible with `WriteImpl::HarnessCallback`.
//!
//! In theory this could be replaced with just making `caller::do_test` into `main`
//! but this might be a bit easier..?
extern {
fn do_test();
}
fn main() {
do_test();
}

@Gankra Gankra added enhancement New feature or request help wanted Extra attention is needed question Further information is requested labels Jun 30, 2024
@Gankra
Copy link
Owner Author

Gankra commented Jul 7, 2024

I'm not sure if this can be part of abi-cafe proper but I would like the test failures to be easy to hang off of a github CI task as an artifacts zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant