Add option to run all tests #53527

Emerentius · 2018-08-20T13:33:16Z

This adds the "--include-ignored" flag to libtest, which allows running ignored and unignored tests in one go.

rust-highfive · 2018-08-20T13:33:26Z

(rust_highfive has picked a reviewer for you, use r? to override)

Emerentius · 2018-08-20T13:37:54Z

I don't know if "--all" is such a good name when we also have flags for "--tests" and "--benches", but I'm pulling blank on alternatives. I'm thinking it should be ok because the most common usage would be cargo test -- --all.

Mark-Simulacrum

A few minor nits, but otherwise this looks good.

Mark-Simulacrum · 2018-08-20T13:37:12Z

src/libtest/lib.rs

@@ -544,7 +552,11 @@ pub fn parse_opts(args: &[String]) -> Option<OptRes> {
        None
    };

-    let run_ignored = matches.opt_present("ignored");
+    let run_ignored = match (matches.opt_present("all"), matches.opt_present("ignored")) {
+        (true, _) => RunIgnored::Yes,


I wonder if it makes sense to error here if we've been passed both; what do you think?

Hmm, maybe. It's often better for humans but worse for automated tools. It's also the more conservative approach that we can always relax later so I'm going for it.

Mark-Simulacrum · 2018-08-20T13:39:42Z

src/libtest/lib.rs

@@ -1869,7 +1887,21 @@ mod tests {
            Some(Ok(o)) => o,
            _ => panic!("Malformed arg in parse_ignored_flag"),
        };
-        assert!((opts.run_ignored));
+        assert!((opts.run_ignored == RunIgnored::Only));


assert_eq and needless inner parentheses

Mark-Simulacrum · 2018-08-20T13:40:31Z

src/libtest/lib.rs

+        ];
+        let opts = match parse_opts(&args) {
+            Some(Ok(o)) => o,
+            _ => panic!("Malformed arg in parse_ignored_flag"),


No need for this panic, we can probably just unwrap?

Yeah. Also, I think this is an error message for a buggy test, not a buggy parse_opts.

Mark-Simulacrum · 2018-08-20T13:43:00Z

@rust-lang/libs - as an insta-stable change asking for FCP (I don't think I can start it).

Cc @rust-lang/dev-tools, not sure if this is under your purview...

SimonSapin · 2018-08-20T13:50:12Z

libtest is semi-private / permanently unstable (except possibly for Bencher), so I don’t think there is a library discussion to be had here.

same behaviour, just shorter

add --all flag to libtest that runs ignored and not ignored tests

Manishearth · 2018-08-20T16:59:33Z

@djrenren on devtools is working on custom test frameworks and probably should have a say on this

djrenren · 2018-08-20T17:32:13Z

This looks really helpful. Two nits:

I'd recommend --force that way this could be used with test, bench, or filtered subset. It also conveys that we'd be overriding something.
Since this is a change to the public-facing interface (which I believe we consider stable?) shouldn't we run this through nightly?

Emerentius · 2018-08-20T20:33:39Z

I associate --force quite strongly, though not exclusively, with operations that are dangerous, inadvisable or rarely needed like rm -f or git {push | pull | clean | checkout } --force

shouldn't we run this through nightly?

What does that mean in this context?

djrenren · 2018-08-20T20:38:55Z

Presumably ignored tests are ignored for a reason. You're explicitly going against the intentions of the author so to me it seems worth of a --force, but maybe I don't have the correct mental model for ignored tests.

What does that mean in this context?

Ah @Mark-Simulacrum mentioned making this an insta-stable change rather than starting as nightly-only then stabilizing later.

Emerentius · 2018-08-20T21:53:33Z

If the author didn't want it to be run ever it would be deleted, commented out or not marked #[test]. There is of course no prescribed reason for which #[ignore] may be used, but realistic cases would be for example very expensive computations or things that access external resources (say, access certain files that may not be setup). In fact, expensive computations is what I've used them for. I've done a quick search through the rust repo. There aren't many occasions, but I've found that the book uses expensive tests as an example and there are two such tests for float printing. A test in cargo's test suite is ignored because it does a web request.
I'm thinking of #[ignore] more like an "off by default"

There's an unstable options flag that supposedly only allows the nightly compiler to use certain flags, but it seems like it really just enforces passing -Zunstable-options prior to using the unstable option. --format json is currently the only unstable option.

SimonSapin · 2018-08-21T13:04:25Z

It isn’t great that the subtly different commands cargo test --all and cargo test -- --all would do very different things. I’d prefer another name, but I don’t really have a suggestion.

Emerentius · 2018-08-21T15:23:40Z

I was thinking something like --ignored (include | skip | only) [default: skip] would be nice, but that's not backwards compatible. Maybe --run-all ?
It may not be obvious with that name that this can still be combined with filters.

nrc · 2018-08-22T01:45:31Z

Seems like a nice thing to have from my perspective. I agree --all is not great. Maybe --test-ignored or something similarly explicit?

fixed

Emerentius · 2018-08-23T17:27:00Z

--include-ignored ?
Should be quite clear, but it's long and imo too close to --ignored

Emerentius · 2018-08-26T16:48:12Z

What's the procedure for moving forward? I can't think think of any better flag names than the ones I've proposed so far.
Will there be an FCP to give others a chance to give input?

Mark-Simulacrum · 2018-08-26T17:57:07Z

I'm going to reassign to r? @nrc since I think the dev tools team is where the decision needs to come from at this point

bors · 2018-09-11T11:26:36Z

⌛ Testing commit f6f3228 with merge e6ec19172a3119598e81f83d9f9a962f78c92c06...

bors · 2018-09-11T12:39:44Z

💔 Test failed - status-appveyor

kennytm · 2018-09-11T14:30:01Z

This has broken a dependency compiletest_rs which clippy's test relies on

[00:55:31] error[E0308]: mismatched types
[00:55:31]    --> C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\compiletest_rs-0.3.13\src\lib.rs:100:22
[00:55:31]     |
[00:55:31] 100 |         run_ignored: config.run_ignored,
[00:55:31]     |                      ^^^^^^^^^^^^^^^^^^ expected enum `test::RunIgnored`, found bool
[00:55:31]     |
[00:55:31]     = note: expected type `test::RunIgnored`
[00:55:31]                found type `bool`
[00:55:31] 
[00:55:32] error: aborting due to previous error

Emerentius · 2018-09-11T18:43:49Z

@kennytm So that's an external crate that calls into rustc internals? I'm a bit confused on what I should do now.
Get a patch ready for compiletest_rs so this can be changed in sync? Some intermediate breakage seems necessary.

Mark-Simulacrum · 2018-09-11T21:54:17Z

You'll probably want to just retry the build in a few days at which point this should land since we don't require clippy to pass normally, just before and during release week. Once this lands compiletest on crates.io can be updated and finally clippy update can land (or just Cargo.lock, perhaps).

kennytm · 2018-09-15T07:53:32Z

@bors retry

bors · 2018-09-15T15:01:58Z

⌛ Testing commit f6f3228 with merge 9f53c87...

Add option to run all tests This adds the "--include-ignored" flag to libtest, which allows running ignored and unignored tests in one go. Closes #50363

bors · 2018-09-15T17:46:41Z

☀️ Test successful - status-appveyor, status-travis
Approved by: nrc
Pushing 9f53c87 to master...

rust-highfive · 2018-09-15T17:48:01Z

📣 Toolstate changed by #53527!

Tested on commit 9f53c87.
Direct link to PR: #53527

💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk, @rust-lang/infra).

@Manishearth

Tested on commit rust-lang/rust@9f53c87. Direct link to PR: <rust-lang/rust#53527> 💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk, @rust-lang/infra). 💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk, @rust-lang/infra).

eddyb · 2018-09-15T19:44:50Z

In the interest of saving time, and breaking as little as possible, I'll add a From<bool> impl for this new enum in #54116, and update compiletest-rs to use From::from.

EDIT: oh we don't own compiletest-rs... I'll revert this PR instead. It really shouldn't have landed so nearly to cutting the RC. Apologies for the inconvenience.

This reverts commit 9f53c87, reversing changes made to cba0fdf.

eddyb · 2018-09-15T22:39:10Z

So I think we should try to do this without breaking compiletest_rs. I'm looking into it right now.
EDIT: okay, so it seems that the usual thing is to just break it and release a patch version - e.g. Manishearth/compiletest-rs#99 - that means we should wait for beta (Rust 2018 RC1) to branch off and then merge this PR again.

Emerentius · 2018-09-26T18:17:27Z

@eddyb Have the waters calmed enough that this can be merged again? Should I open a new PR?

eddyb · 2018-09-28T13:53:27Z

@Emerentius Yes, sorry to delay this and keep you waiting! RC1 went out about a week ago.

Add option to run all tests, again This is a repeat of #53527, which had to be reverted to land #54116. It will break clippy until `compiletest-rs` can be updated and I believe we're closing on a new release date, so this may need to be delayed again until after 1.30 is out (?) Closes #50363 again

rust-highfive assigned Mark-Simulacrum Aug 20, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 20, 2018

Mark-Simulacrum previously requested changes Aug 20, 2018

View reviewed changes

Emerentius force-pushed the test_all branch from 184d078 to ffa6c2d Compare August 20, 2018 14:14

This comment has been minimized.

Sign in to view

Emerentius force-pushed the test_all branch from ffa6c2d to 8c16a97 Compare August 20, 2018 14:50

This comment has been minimized.

Sign in to view

Emerentius added 2 commits August 20, 2018 18:29

refactor filter_tests

435b0ab

same behaviour, just shorter

add option to run all tests

9adf26a

add --all flag to libtest that runs ignored and not ignored tests

Emerentius force-pushed the test_all branch from 8c16a97 to 9adf26a Compare August 20, 2018 16:30

rust-highfive assigned nrc and unassigned Mark-Simulacrum Aug 26, 2018

nrc added I-nominated T-dev-tools Relevant to the dev-tools subteam, which will review and decide on the PR/issue. labels Aug 26, 2018

bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Sep 11, 2018

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 11, 2018

kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 11, 2018

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 15, 2018

bors added a commit that referenced this pull request Sep 15, 2018

Auto merge of #53527 - Emerentius:test_all, r=nrc

9f53c87

Add option to run all tests This adds the "--include-ignored" flag to libtest, which allows running ignored and unignored tests in one go. Closes #50363

bors merged commit f6f3228 into rust-lang:master Sep 15, 2018

eddyb mentioned this pull request Sep 15, 2018

rustc_resolve: allow only core, std, meta and --extern in Rust 2018 paths. #54116

Merged

eddyb added a commit to eddyb/rust that referenced this pull request Sep 15, 2018

Revert "Auto merge of rust-lang#53527 - Emerentius:test_all, r=nrc"

bde0a54

This reverts commit 9f53c87, reversing changes made to cba0fdf.

eddyb mentioned this pull request Sep 15, 2018

libtest: provide a way to run *all* tests #50363

Closed

Emerentius mentioned this pull request Oct 13, 2018

Add option to run all tests, again #55053

Merged

ehuss mentioned this pull request Oct 24, 2019

libtest --include-ignored nightly-gating is confusing #65770

Closed

Add option to run all tests #53527

Add option to run all tests #53527

Conversation

Emerentius commented Aug 20, 2018 • edited Loading

rust-highfive commented Aug 20, 2018

Emerentius commented Aug 20, 2018

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Mark-Simulacrum Aug 20, 2018

Choose a reason for hiding this comment

Emerentius Aug 20, 2018 • edited Loading

Choose a reason for hiding this comment

Mark-Simulacrum Aug 20, 2018

Choose a reason for hiding this comment

Mark-Simulacrum Aug 20, 2018

Choose a reason for hiding this comment

Emerentius Aug 20, 2018

Choose a reason for hiding this comment

Mark-Simulacrum commented Aug 20, 2018

SimonSapin commented Aug 20, 2018

This comment has been minimized.

This comment has been minimized.

Manishearth commented Aug 20, 2018

djrenren commented Aug 20, 2018

Emerentius commented Aug 20, 2018

djrenren commented Aug 20, 2018

Emerentius commented Aug 20, 2018 • edited Loading

SimonSapin commented Aug 21, 2018

Emerentius commented Aug 21, 2018 • edited Loading

nrc commented Aug 22, 2018

Emerentius commented Aug 23, 2018 • edited Loading

Emerentius commented Aug 26, 2018

Mark-Simulacrum commented Aug 26, 2018

bors commented Sep 11, 2018

bors commented Sep 11, 2018

kennytm commented Sep 11, 2018

Emerentius commented Sep 11, 2018

Mark-Simulacrum commented Sep 11, 2018

kennytm commented Sep 15, 2018

bors commented Sep 15, 2018

bors commented Sep 15, 2018

rust-highfive commented Sep 15, 2018

eddyb commented Sep 15, 2018 • edited Loading

eddyb commented Sep 15, 2018 • edited Loading

Emerentius commented Sep 26, 2018

eddyb commented Sep 28, 2018

Emerentius commented Aug 20, 2018 •

edited

Loading

Emerentius Aug 20, 2018 •

edited

Loading

Emerentius commented Aug 20, 2018 •

edited

Loading

Emerentius commented Aug 21, 2018 •

edited

Loading

Emerentius commented Aug 23, 2018 •

edited

Loading

eddyb commented Sep 15, 2018 •

edited

Loading

eddyb commented Sep 15, 2018 •

edited

Loading