Discussion: run specific test - test.only

[bartlomieju]

I got some test that is failing and I'd want to debug it, so when I run my test suite I want to focus some tests without needing to comment out my test.ts file. It boosts productivity significantly and would be nice to have it in standard lib.

This pattern is common in Node.js testing frameworks:

• AVA - test.only
• Mocha - describe.only/it.only
• Jest - test.only/it.only/fit

Proposed API:
test.only(t: TestDefinition | TestFunction): void - when there's at least one usage of this method only those test cases are run during runTests

EDIT: Things to consider:

• how it should play with filter in testing module?

[zekth]

In addition i'm also using the jest cli parameter -t like this:

jest -t name-of-spec

This is nice to debug. Also in a homemade framework we added something like glob on spec names. For example:

testcli -f feature-module-* // this will test all the spec files matching this glob parameter

[hayd]

Test cli is waiting on import(), would be able glob on filename and test.
#193

[zekth]

@hayd once the import() is implemented we can change the filter part of testing module to handle regex and glob too?
https://github.com/denoland/deno_std/blob/master/testing/mod.ts#L210

[chiefbiiko]

An alternative to the test.only pattern could be, adjusting this interface to:

/** Defines options for controlling execution details of a test suite. */
export interface RunOptions {
  parallel?: boolean;
  only?: RegExp;
  skip?: RegExp;
}

Semantics
Any test case that matches the only pattern and does not match the skip pattern is run.
Any test case that does not match the only pattern or does match the skip pattern is not run.

That would allow test case filtering like:

runTests({ only: /caseA|caseB/ });
runTests({ only: /caseA|caseB/, skip: /caseC/ });
runIfMain(import.meta, { only: /caseA|caseB/ });
runIfMain(import.meta, { skip: /caseC/ });

This filtering approach is already implemented in benching.

Characteristics

• only and skip can reference multiple test names with a simple regex (/a|b|c/)
• only and skip can both be specified
• changing test case filtering is done in exactly one place in the file (options of the run* executor)
• uncommon

What y'all think about this?

[zekth]

Is the regexp pattern checking the filename of the import? If so it's ok for me and we can glob it too like @hayd suggested

[hayd]

There's a filter regex already, having runTests look at a named --filter arg, and passing it as the filter, seems reasonable to me.

i.e. so can do deno test.ts --filter specificTest

[bartlomieju]

@chiefbiiko your solution still requires to go to entry point of tests and place a filter. My initial idea was to just append .only to particular test case that I'm working on.

We might consider adding test.skipIf as well:

test.skipIf(Deno.platform === "win", function myTest() { ... })

[zekth]

Could be nice doing something like this in tests scripts :

Test.skipIf(expression, () => {
  Test(function testToBeSkippedIf() {
   })
   Test(function testToBeSkippedIf2() {
   })
})

[hayd]

One alternative is skip() that raises a SkipTestError which we catch/handle in runTests.

So code would be:

test(function Foo(){
  if (Deno.platform == 'win') { skip() }
  ...
})

Edit: can do pending() in the same way

[dsseng]

@bartlomieju @zekth @hayd @chiefbiiko what's decision about this one? I'd like to do this

[harrysarson]

Based on my own experience of using test.only when developing a pull request and forgetting to remove it when submitting the patch...

It would be really good if using test.only would cause the test runner to exit with a non zero exit code so that CI services will pick up accidental additions of test.only.

[zekth]

I don't think it's a good idea to exit with a non zero error code because in some cases you want to use test.only depending on the env so it will be a mess for the CI to handle this.