Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to stop after some number of failed tests #114

Closed
3 tasks done
nfachan opened this issue Jan 9, 2024 · 3 comments
Closed
3 tasks done

Add ability to stop after some number of failed tests #114

nfachan opened this issue Jan 9, 2024 · 3 comments
Assignees
Labels
enhancement New feature or request test-runners Issues that affect all test runners (cargo-maelstrom, maelstrom-pytest, maelstrom-to-test, etc.).
Milestone

Comments

@nfachan
Copy link
Collaborator

nfachan commented Jan 9, 2024

TAC

  • This should be a configuration value. A user could want to set this in their configuration.
  • The configuration value should be called stop-after. I'm open to other suggestions. I prefer imperative verbs like "stop", but I'll admit it's not obvious what we're stopping after. We could consider something like max-failures or failure-limit.
  • Upon reaching the Nth failure, the test runner should immediately print a summary and exit. This will stop any pending tests. We don't want to wait for outstanding tests.
  • The summary message should at a minimum indicate that some tests weren't run. Ideally, we would print more information, but the fact that we may not have built all test binaries complicates things. If we really wanted to go for extra credit, we could print the number of tests that we know haven't been executed, and then possibly add the number of tests that we think will be run, once those test binaries were built.
  • We should ensure that cargo test --no-run will immediately exit in such scenarios. Presumably it will get a SIGPIPE eventually, but we may want to accelerate the process.

Definition of Done

  • CHANGELOG.md updated.
  • Documentation in doc/book/head updated.
  • Issue is in the correct milestone.
@nfachan nfachan added enhancement New feature or request cargo-maelstrom labels Jan 9, 2024
@nfachan nfachan added this to the 0.4.0 milestone Jan 9, 2024
@nfachan
Copy link
Collaborator Author

nfachan commented Jan 9, 2024

This is complicated by build failures. We should think this through a little bit.

cargo test and cargo nextest have --fail-fast and --no-fail-fast.

@nfachan
Copy link
Collaborator Author

nfachan commented Jan 10, 2024

Yeah, that's definitely a complexity. I'm wondering if we want to have separate options for the two, and then maybe a convenience option that can control both of them at the same time. When cargo metest runs cargo, it could pass --fail-fast or --no-fail-fast, right?

@nfachan nfachan modified the milestones: 0.4.0, Launch Jan 11, 2024
@nfachan nfachan modified the milestones: Launch, 0.6.0 Feb 8, 2024
@nfachan nfachan modified the milestones: 0.6.0, 0.7.0, Launch Mar 29, 2024
@nfachan nfachan removed this from the Soon milestone Jul 15, 2024
@nfachan nfachan added this to the 0.12.0 milestone Jul 31, 2024
@nfachan
Copy link
Collaborator Author

nfachan commented Jul 31, 2024

I don't think we have to worry about build failures. The current behavior is to immediately stop when the build fails, which is what I think we want to continue to do.

@nfachan nfachan added test-runners Issues that affect all test runners (cargo-maelstrom, maelstrom-pytest, maelstrom-to-test, etc.). and removed cargo-maelstrom labels Jul 31, 2024
@bobbobbio bobbobbio self-assigned this Aug 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request test-runners Issues that affect all test runners (cargo-maelstrom, maelstrom-pytest, maelstrom-to-test, etc.).
Projects
None yet
Development

No branches or pull requests

2 participants