Add ability to run solo tests

[ltackmann]

something like test(solo:true) even better a Test annotation that can be selected from the dart editor to select that only a single test should run

[nex3]

This functionality was very intentionally moved to being handled via command-line flags. Modifying test code to support choosing which tests to run was always a hack; it only existed to work around the inability to consistently parse command-line arguments in tests. It makes it really easy to accidentally check in a change that silently disables a bunch of your tests, and really hard for tooling to select which tests to run.

[ltackmann]

This makes sense. However having to create a new test runner in the dart editor with the correct command line flags evertime I want to execute a single test is a bit tedious. Perhaps creating a @test and @testgroup annotation for tests would allow some IDE tooling to select just a single test or group of tests for execution (like what is done in TestNG and JUnit).

[nex3]

It seems way easier for IDE tooling to pass an argument to the test runner than to modify the test code itself.

[yjbanov]

Reopening the issue. I'm fine with allowing isolating tests from the command line, but disallowing isolating tests in the test code is a big big loss in developer productivity. Typing strings on the command line means a lot more typing and also either having to copy the text to paste (which involves many more mouse clicks and keystrokes) or memorizing and carefully entering the string/regex yourself. groups existed to allow you to organize related tests in nested groups so you can easily isolate them with fine control over granularity of the functional area of your code you are testing. You could choose multiple groups at different levels and on different branches of your group hierarchy. This was super-useful in unittest (as well as guinness). For example, I could isolate a unit test for a class and a related integration test elsewhere in the test hierarchy. This ability is pretty much destroyed with this change. Groups now are there for visual organization and perhaps a bit of string deduplication, and not much more.

[nex3]

@yjbanov This still isn't a feature we plan to add. I understand why it's somewhat easier under some circumstances, but it's also very error-prone and largely redundant with existing functionality. You're welcome to write a wrapper (or eventually a plug-in) that supports this, but there's not going to be direct support.

You could choose multiple groups at different levels and on different branches of your group hierarchy. This was super-useful in unittest (as well as guinness). For example, I could isolate a unit test for a class and a related integration test elsewhere in the test hierarchy. This ability is pretty much destroyed with this change.

You can still do this via something like -n 'test 1|test 2'. If you'd like, feel free to file an issue to add support for multiple -n flags as well.

Groups now are there for visual organization and perhaps a bit of string deduplication, and not much more.

These are legitimately useful things, but they're not the extent of groups' utility. Groups are used for defining configuration or setUp/tearDown for specific sets of tests, and it's pretty easy to select tests in a particular group with -n '^group name'.

[nex3]

I should also mention that you'll be able to roll your own solo testing to some degree once #16 is implemented. If you write something like test(..., tag: "solo") you'll then be able to run only that test (or group, or collection of tests) with -t solo.

[yjbanov]

I agree that's a cleaner API, but it still does not provide sufficient ergonomics for fast-paced test-driven workflows such as the one used by Angular and the users of Angular. We prefer to run our tests continuously, so you do not even have to switch away from your code editor. You pick a collection of tests you want to run (via isolating them in the code, where your focus is), save file and the test runner (sensing file system changes) reruns everything automatically. This allows very fast code/test iterations (on the order of seconds per iteration). Having to go to the command-line and to enter flags would slow down the process significantly, as would having to enter tag: "solo".

I suppose we should be able to wrap package:test API into something more ergonomic, but I'm sad to see it not supported out-of-the-box.

BTW, the problem of the risk of submitting isolated tests can be easily solved by the command-line flag to the test runner --enable-isolated-tests, which would be false by default.

[nex3]

A continuous test runner like the one you're describing doesn't exist yet; once it does it will likely be in a different package that builds on the test infrastructure rather than being baked into test itself. It should be straightforward for that package to add additional features that fit its particular ergonomical needs. If it can't do it on its own, we can revisit this to see if we can add the necessary support to test without making it worse for people not using the continuous runner.

For now, though, this package only supports batch-mode running. In batch mode a combination of -n and tagging works very well—it doesn't accidentally disable tests or require an additional option, and tagging produces sets of tests to focus on that are persistent over time.

[pq]

Re-opening to give this yet another nudge. I understand the principle behind the choice not to implement this but in practice it's a CONSTANT drag. I'm w/ @yjbanov on the plea for better ergonomics out of the box.