Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Random test ordering for cargo test #85440

Open
stchris opened this issue May 18, 2021 · 9 comments
Open

Random test ordering for cargo test #85440

stchris opened this issue May 18, 2021 · 9 comments
Labels
A-libtest Area: `#[test]` / the `test` library C-feature-request Category: A feature request, i.e: not implemented / a PR. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@stchris
Copy link
Contributor

stchris commented May 18, 2021

Describe the problem you are trying to solve
Sometimes tests depend on each other and one of the possible counter measures is to shuffle the order in which tests are run.

Describe the solution you'd like
I would like cargo test to take a --random flag, which would shuffle the test order. Ideally it would generate and print the seed used (a u64 integer). This could be used to debug situations, maybe with a corresponding --random-seed <seed> parameter (which would imply --random).

Notes
I had a look at the codebase and I think I identified the place where the sorting currently happens (https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/cargo_test.rs#L67). I'd be happy to try to contribute a PR.
@ehuss
Copy link
Contributor

ehuss commented May 18, 2021

Just to clarify, are you referring to individual #[test] annotated functions, or integration test executables?

@stchris
Copy link
Contributor Author

stchris commented May 18, 2021

Just to clarify, are you referring to individual #[test] annotated functions, or integration test executables?

I did mean #[test] annotated functions. I realize that I might be looking at the wrong code for doing that.

@ehuss ehuss transferred this issue from rust-lang/cargo May 18, 2021
@ehuss
Copy link
Contributor

ehuss commented May 18, 2021

Ah. I have transferred this to the rust-lang/rust repository where the libtest harness lives. The code for the test harness lives in the library/test directory.

@jonas-schievink jonas-schievink added A-libtest Area: `#[test]` / the `test` library C-feature-request Category: A feature request, i.e: not implemented / a PR. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. labels May 18, 2021
@stchris
Copy link
Contributor Author

stchris commented May 18, 2021

I see. Thank you @ehuss !

@smoelius
Copy link
Contributor

FWIW, Go recently acquired such a feature (maybe that inspired this issue?):
https://go-review.googlesource.com/c/go/+/310033/
https://www.reddit.com/r/golang/comments/nktggq/as_of_go_117_youll_be_able_to_shuffle_the/

@smoelius
Copy link
Contributor

@stchris Have you tried implementing this?

smoelius added a commit to smoelius/rust that referenced this issue Sep 30, 2021
@smoelius
Implement rust-lang#85440
fa23d4f
Manishearth added a commit to Manishearth/rust that referenced this issue Oct 7, 2021
@Manishearth
Rollup merge of rust-lang#89082 - smoelius:master, r=kennytm
827e9b2 
Implement rust-lang#85440 (Random test ordering)

This PR adds `--shuffle` and `--shuffle-seed` options to `libtest`. The options are similar to the [`-shuffle` option](https://github.com/golang/go/blob/c894b442d1e5e150ad33fa3ce13dbfab1c037b3a/src/testing/testing.go#L1482-L1499) that was recently added to Go.

Here are the relevant parts of the help message:
```
        --shuffle       Run tests in random order
        --shuffle-seed SEED
                        Run tests in random order; seed the random number
                        generator with SEED
...
By default, the tests are run in alphabetical order. Use --shuffle or set
RUST_TEST_SHUFFLE to run the tests in random order. Pass the generated
"shuffle seed" to --shuffle-seed (or set RUST_TEST_SHUFFLE_SEED) to run the
tests in the same order again. Note that --shuffle and --shuffle-seed do not
affect whether the tests are run in parallel.
```
Is an RFC needed for this?
@Manishearth Manishearth mentioned this issue Oct 7, 2021
Closed
Manishearth added a commit to Manishearth/rust that referenced this issue Oct 8, 2021
@Manishearth
Rollup merge of rust-lang#89082 - smoelius:master, r=kennytm
8f254a7 
Implement rust-lang#85440 (Random test ordering)

This PR adds `--shuffle` and `--shuffle-seed` options to `libtest`. The options are similar to the [`-shuffle` option](https://github.com/golang/go/blob/c894b442d1e5e150ad33fa3ce13dbfab1c037b3a/src/testing/testing.go#L1482-L1499) that was recently added to Go.

Here are the relevant parts of the help message:
```
        --shuffle       Run tests in random order
        --shuffle-seed SEED
                        Run tests in random order; seed the random number
                        generator with SEED
...
By default, the tests are run in alphabetical order. Use --shuffle or set
RUST_TEST_SHUFFLE to run the tests in random order. Pass the generated
"shuffle seed" to --shuffle-seed (or set RUST_TEST_SHUFFLE_SEED) to run the
tests in the same order again. Note that --shuffle and --shuffle-seed do not
affect whether the tests are run in parallel.
```
Is an RFC needed for this?
@Manishearth Manishearth mentioned this issue Oct 8, 2021
Closed
workingjubilee added a commit to workingjubilee/rustc that referenced this issue Oct 8, 2021
@workingjubilee
Rollup merge of rust-lang#89082 - smoelius:master, r=kennytm
37f17bc 
Implement rust-lang#85440 (Random test ordering)

This PR adds `--shuffle` and `--shuffle-seed` options to `libtest`. The options are similar to the [`-shuffle` option](https://github.com/golang/go/blob/c894b442d1e5e150ad33fa3ce13dbfab1c037b3a/src/testing/testing.go#L1482-L1499) that was recently added to Go.

Here are the relevant parts of the help message:
```
        --shuffle       Run tests in random order
        --shuffle-seed SEED
                        Run tests in random order; seed the random number
                        generator with SEED
...
By default, the tests are run in alphabetical order. Use --shuffle or set
RUST_TEST_SHUFFLE to run the tests in random order. Pass the generated
"shuffle seed" to --shuffle-seed (or set RUST_TEST_SHUFFLE_SEED) to run the
tests in the same order again. Note that --shuffle and --shuffle-seed do not
affect whether the tests are run in parallel.
```
Is an RFC needed for this?
@workingjubilee workingjubilee mentioned this issue Oct 8, 2021
Merged
bors added a commit to rust-lang-ci/rust that referenced this issue Oct 8, 2021
@bors
Auto merge of rust-lang#89659 - workingjubilee:rollup-0vggc69, r=work…
3013b26 
…ingjubilee

Rollup of 8 pull requests

Successful merges:

 - rust-lang#87918 (Enable AutoFDO.)
 - rust-lang#88137 (On macOS, make strip="symbols" not pass any options to strip)
 - rust-lang#88772 (Fixed confusing wording on Result::map_or_else.)
 - rust-lang#89025 (Implement `#[link_ordinal(n)]`)
 - rust-lang#89082 (Implement rust-lang#85440 (Random test ordering))
 - rust-lang#89288 (Wrapper for `-Z gcc-ld=lld` to invoke rust-lld with the correct flavor)
 - rust-lang#89476 (Correct decoding of foreign expansions during incr. comp.)
 - rust-lang#89622 (Use correct edition for panic in [debug_]assert!().)

Failed merges:

r? `@ghost`
`@rustbot` modify labels: rollup
@hanbollar
Copy link

hiya ~ @ehuss / @smoelius / @stchris is this still planned? would be great to use

@smoelius
Copy link
Contributor

Hi, @hanbollar. This is implemented, but only for nightly.

I run it regularly, actually.

@hanbollar
Copy link

Hi, @hanbollar. This is implemented, but only for nightly.

I run it regularly, actually.

o perf! thx for the info @smoelius

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-libtest Area: `#[test]` / the `test` library C-feature-request Category: A feature request, i.e: not implemented / a PR. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
testing-devex backlog
Status: No status
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ehuss @stchris @jonas-schievink @hanbollar @smoelius