Random test ordering for cargo test

[stchris]

Describe the problem you are trying to solve
Sometimes tests depend on each other and one of the possible counter measures is to shuffle the order in which tests are run.

Describe the solution you'd like
I would like cargo test to take a --random flag, which would shuffle the test order. Ideally it would generate and print the seed used (a u64 integer). This could be used to debug situations, maybe with a corresponding --random-seed <seed> parameter (which would imply --random).

Notes
I had a look at the codebase and I think I identified the place where the sorting currently happens (https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/cargo_test.rs#L67). I'd be happy to try to contribute a PR.

[ehuss]

Just to clarify, are you referring to individual #[test] annotated functions, or integration test executables?

[stchris]

Just to clarify, are you referring to individual #[test] annotated functions, or integration test executables?

I did mean #[test] annotated functions. I realize that I might be looking at the wrong code for doing that.

[ehuss]

Ah. I have transferred this to the rust-lang/rust repository where the libtest harness lives. The code for the test harness lives in the library/test directory.

[stchris]

I see. Thank you @ehuss !

[smoelius]

FWIW, Go recently acquired such a feature (maybe that inspired this issue?):
https://go-review.googlesource.com/c/go/+/310033/
https://www.reddit.com/r/golang/comments/nktggq/as_of_go_117_youll_be_able_to_shuffle_the/

[smoelius]

@stchris Have you tried implementing this?