Unable to add 'only' or 'skip' to child tests

[Jameskmonger]

I can do this:

test.only('hi', t => {
    t.pass('test passed');
    t.end();
});

But I can't do this:

test('hi', t => {

    t.test('this test will not run', s => {
        s.fail('i should not have been executed');
        s.end();
    });

    t.test.only('this test will run', s => {
        s.pass('i should have been executed');
        s.end();
    });
});

At least, the TypeScript typings file doesn't support it. Not sure if it would let me do it in plain JS.

[ljharb]

I suspect it would require using ES5 accessors to work properly. However, I believe you can do:

test('hi', t => {
    t.test('this test will not run', s => {
        s.fail('i should not have been executed');
        s.end();
    });

    t.test('this test will run', { only: true }, s => {
        s.pass('i should have been executed');
        s.end();
    });
});

[dallonf]

Just tried this - it seems to work for skip but not only.

[ljharb]

ok - that seems like it should maybe be an option.

Would t.test.only only run that test in the entire suite? or would it run only that test out of t's tests?

[dallonf]

My intuition is that it should only run that test in the whole suite - I typically use it as a temporary debugging tool to let me focus on one particular test. Unsure what other peoples' use cases for only is, though.

[ljharb]

Makes sense.

I'd be open to a PR that added the { only: true } option, as well as t.test.only - both seem reasonable.

[jtlapp]

I glanced at the relevant code, having gotten myself pretty familiar with tape. This would be pretty easy to do if you restrict the scope of a nested only to the containing test. It would be a bit of a nightmare to reconfigure the code to treat nested tests on par with outer tests to resolve the only and skip designations at their various levels. You'd probably have to restructure the code to handle nested tests and non-nested tests identically except at time of definition. The interactions of only and skip are already a bear for the outer scope alone.

[tjconcept]

I originally assumed t.test was actually the same class as test albeit with some reference/reporting to the parent, but quickly learned that e.g. .only does not exist.

Would t.test.only only run that test in the entire suite? or would it run only that test out of t's tests?

The least surprising to me would be for it to affect only the same-level tests such that the rest of the suite would still run, unless the parent too has the only flag.

[Raynos]

The only feature is really nuanced because of the order of execution.

Basically sub tests and tests run in tree order. You cannot only a subtest. You’d have to only the outer test and the sub test.

The global list of all sub tests is dynamic and gets added to as each tests run. An only in subtest for test 3 would only get called after test 1 and 2 complete.

There’s literally no good way around this.

I’d recommend adding an only method to subtest that throws a not support exception and a reference to the documentation.

I had considered deleting the entire subtest feature 6 years ago because it’s complicated and full of gotchas.

[Raynos]

Other frameworks support subtest only by having a test runner cli aka ( Ava or jest ) and running a regex on the test file before executing it.

Basically use static analysis to find all references to only before running any tests.

Such a feature won’t work for tape

[tjconcept]

Other frameworks support subtest only by having a test runner cli aka ( Ava or jest ) and running a regex on the test file before executing it.

Basically use static analysis to find all references to only before running any tests.

That sounds horrible.

Basically sub tests and tests run in tree order. You cannot only a subtest. You’d have to only the outer test and the sub test.

I don't understand the problem. If you want to only run a subtest, set only on the parent and the subtest. All tests are registered on each level before any of them are executed.

I had considered deleting the entire subtest feature 6 years ago because it’s complicated and full of gotchas.

Isn't it possible to make it a proper parent/child relationship of equal instances? What is keeping t.test from being the same as test albeit with output routed to the parent?

[Raynos]

That’s a reasonable suggestion

Make t.test.only throw if the outer test for t is not test.only

You can only use nested only test cases in an only test case

[tjconcept]

Make t.test.only throw if the outer test for t is not test.only

You can only use nested only test cases in an only test case

Why that constraint? I wouldn't expect an only on a subtest to affect any other tests or their subtests but the "sisters" to the one marked.

[ljharb]

The nested "only" still has to invoke the entirety of its "parents" in order to work properly - that might be unexpected/surprising if the parent tests aren't all marked "only".

The counterpoint is that the UX of having to make multiple changes to set an "only" would be abjectly terrible.

[tjconcept]

An alternative approach, allowing only to have a "global impact", could be a new construct (test.group) to simply group a bunch of tests together. All grouped tests would inherit flags from the test.group({..}, t. => ..). The groups could then be immediately invoked to register all "subtests" alongside the rest of the tests.

[ljharb]

A possible alternative to "only" is something like "filter" - it would be a (CLI arg/env var) pattern to apply to test descriptions, and while every test would execute, only results within tests that matched the filter would be displayed or count towards the total or the exit code.

[tjconcept]

Indeed, but I don't think they serve the exact same purpose. I often use only during debugging to avoid stepping through everything and to filter away additional logging from the rest.