cargo test: Don't re-run tests that already passed

[alecmocatta]

Problem

I have a large workspace in which there's a crate with several long-running property tests. This makes cargo test --workspace take unreasonably long in the common case where those property tests haven't been touched.

Proposed Solution

Cache the test results, so that cargo test --workspace can complete quickly in the common case where long-running tests haven't been touched.

Notes

Related work:

• Caching of test results has already been impl'd for compiletest: rust-lang/rust@c8e0d04 rustbuild: Don't re-run tests suites that already passed rust#36385

[epage]

I suspect this is more of a libtest concern but there is the issue of cargo running all of the libtest binaries.

[alecmocatta]

For my use case, the libtest binary is a fine level of granularity to cache the results of. Caching individual #[test]s within a libtest binary seems more fragile?

[weihanglo]

IIRC, cargo test is not aware of external environment changes, such as environment variables or files. It might be abit risky to cache and skip previously passed tests as cargo cannot 100% sure about what has been changed.

I know that in JavaScript world, a famous test framework jest gets a shiny watch mode that let you re-run failure tests or a portion of filtered tests in the same interactive session. However, even jest re-runs all tests if you re-invoke the command, so I doubt that cargo should skip passed tests across invocations.

Anyway, this is great advice. The field of test framework in Rust is still young. Maybe there are already some similar tools in Rust. Really looking forwards to some external-command doing what jest does :)

[epage]

For other prior art, pytest has

  --lf, --last-failed   rerun only the tests that failed at the last run (or
                        all if none failed)
  --ff, --failed-first  run all tests, but run the last failures first.
                        This may re-order tests and thus lead to repeated
                        fixture setup/teardown.
  --nf, --new-first     run tests from new files first, then the rest of the
                        tests sorted by file mtime

See https://docs.pytest.org/en/7.1.x/reference/reference.html#command-line-flags

Users opt-in to running failed tests so invalidation isn't a concern.

[leighmcculloch]

cargo test is not aware of external environment changes, such as environment variables or files

Is there a way to give it access to these things?

+1 This feature. In go test test results and output are cached and reads of env vars and files are tracked so that just those that are read affect the cache. It makes a huge difference to testing large or projects and monorepos.

go test caches successful package test results to avoid unnecessary repeated running of tests. When the result of a test can be recovered from the cache, go test will redisplay the previous output instead of running the test binary again...

The rule for a match in the cache is that the run involves the same test binary and the flags on the command line... Tests that open files within the package's source root (usually $GOPATH) or that consult environment variables only match future runs in which the files and environment variables are unchanged.

Ref: https://pkg.go.dev/cmd/go#hdr-Test_packages

[weihanglo]

Probably kinda a duplicate of #6826. This has got more discussions so close that in favor of this.

I'd say the relationship between libtest and cargo-test is a bit unclear and there is no obvious progress so far. Some experiments (https://github.com/epage/pytest-rs) and well-developed alternative (https://nexte.st/) are out there. You may want to take a look.

[alecmocatta]

Thanks @weihanglo. For reference here's the corresponding issue in Nextest nextest-rs/nextest#174

[casey]

One way to support this would to add a flag to cargo test which would save a JSON report of passed and failed tests. This would allow users to write tools which only run the failed tests.

[epage]

While T-testing-devex is still establishing a plan, my expectation for how something like this would work: once libtest has stable json output and cargo leverages it for taking on more responsibilities of libtest, third-party test runners would experiment with what can/should be done in this space and as the design space is explored, we can consider what to include within cargo test.

[dnbln]

For more prior art, I would like to throw in a shameless plug: over the past year I've worked on cargo-difftests, which attempts to tackle the issue of invalidation (although it pretty much always assumes the tests ran successfully). It uses LLVM coverage information to determine which tests have been "touched," although there might be some other alternatives to this approach which I've not tried to explore (yet?).

[extradosages]

This is a great idea. Strong feature for TDD.

[AlexandreCassagne]

Does the compiler support tracing which symbols/functions were invalidated?

[gtrak]

ocaml/dune does this ocaml/dune#4390 (comment)