Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DX] Prevent cargo test from working, point users towards cargo nextest #8691

Open
nagisa opened this issue Mar 7, 2023 · 1 comment
Open
Labels
C-housekeeping Category: Refactoring, cleanups, code quality

Comments

@nagisa
Copy link
Collaborator

nagisa commented Mar 7, 2023

CI currently enforces that cargo nextest rather than cargo test passes. Due to the differences between how these two runners run tests and test harnesses, it is straightforward to end up in a situation where various invocations of cargo test just outright fail to pass (due to e.g. global state details) despite largely always-green HEAD.

While these failures may be an indication of poorly written test, there is little reason to actually spend time on maintaining cargo test in a working condition when the primary motivation behind moving away from cargo test in the CI was to get rid of flakiness (#8207). So clearly we already declared this a lost battle and probably won't return back to fighting it unless something changes drastically.

It is still a terrible developer experience when they learn about this after they have spent 3 hours debugging test failures. We should figure out if and how we could make cargo test invocations fail loudly, quickly, every time, and with a helpful message pointing them at cargo nextest.

@nagisa nagisa added the C-housekeeping Category: Refactoring, cleanups, code quality label Mar 7, 2023
@nagisa
Copy link
Collaborator Author

nagisa commented Mar 9, 2023

We have a way to check if the tests are being run under nextest. See here for context, the important part being…

let Ok(nextest) = std::env::var("NEXTEST") else { return };

I think the most sure way to get a check like this to run in most cases would be to maintain some life-before-main in some crate that's used universally across all crates with tests and place the check there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-housekeeping Category: Refactoring, cleanups, code quality
Projects
None yet
Development

No branches or pull requests

1 participant