tink_unittest support

[kevinresol]

I would like to make this support tink_unittest. What are the requirements?

[AlexHaxe]

Basically two things:

1. you need a way to report test results to testadapter.data.TestResultData. Basically calling addPass, addFail, etc. for every test function / case
2. you need a way to filter tests using testadapter.data.TestFilter.shouldRunTest(cls, name)

Of course both should only happen when -lib test-adapter is used (Haxe Test Explorer appends it automatically when run from VSCode).

For munit, utest and haxe.unit we (well it was @Gama11's brilliant idea) where able to make it work without modifications and by injecting everything "from the outside". So users don't need to modfiy any source code. It even works on projects that don't use -lib test-adapter in their buildTest.hxml (with the caveat, that you have to run all tests from Test Explorer first)

[Gama11]

Things are a bit in flux right now. The extension now has a samples directory for the different unit testing frameworks, and I just updated the usage guide: https://github.com/vshaxe/haxe-test-adapter#usage

[AlexHaxe]

I guess for tink_unittest it's a bit complicated, because it already uses a few macros.

I think one approach would be to overwrite TestCase.execute to record test results (rename it with a build macro (using Compiler.addMetadata) to infuse a new execute function with test adapter functionality).
One might even filter by just not executing a testcase, but it still has to return something and it probably ends up in a report.
But naturally you will have a better solution, because I'm not really familiar with tink_unittest.

[kevinresol]

For reporting I guess it would be best to do so in a Reporter

As for filtering I think somehow it needs to feed a full list to the adapter?

[AlexHaxe]

yes, a reporter can feed test results into test adapter. I am not sure if it is possible to add one on the fly so that a compilation with -lib test-adapter prints out its usual report on stdout but also writes test results for test adapter.

Filtering requires a call to testadapter.data.TestFilter.shouldRunTest(clazz, method) for every test case, which tells you whether to include or skip it for an actual run. That of course should happen only when -lib test-adapter is set. It should be safe to run it inside a macro, if that helps.

I think after a busy day yesterday things have stabilised a bit and quite a few bugs where fixed. The API should not change much anymore.

You might want to look into testadapter.Macro to make sure line numbers get recorded (there is a regex filtering class names). You might also look at one of our macros injecting test adapter functions into munit / utest / haxe.unit, to see if you can write similar for tink_unittest / tink_testrunner.

[edit]and it should run on Haxe 3 and 4[/edit]

[AlexHaxe]

After working on buddy support, I looked at tink_unittest again and I think it would work in a similar fashion.

I would basically add a build macro to tink.testrunner.Runner to replace run and runCase.
In new run I would instantiate a new test adapter controlled reporter, that proxies the one passed in from outside and collects test results.
In new runCase I would apply filtering and simply return null in case of no match (null should be fine, because the return value seems to go unused).

It's only a short summary and I might miss a few obstacles, but that's how I think it might work.

[AlexHaxe]

@kevinresol just for clarification: are you still working / planning to work on a tink_unittest / tink_testrunner integration and do you need support?

[kevinresol]

Sorry but I just don't have time to work on it in short term.

[AlexHaxe]

Don't worry, I might have time to work on it later today.
Once we have something that works we can always improve it in a later revision.

[AlexHaxe]

I have initial tink_testrunner support running.

I had to disable our error testcase because it throws an exception which tink_testrunner doesn't seem to catch and I can see no Error result, so I guess the framework doesn't deal well with exceptions. How do you get an error to show up - BasicReporter always shows 0 Error during my tests?

The same goes for ignored tests, you can @:exclude them, but there is no report on them, is there a way to have ignored tests reported as Ignored?

For filtering test cases I return

Future.async(function(cb) {
    cb({info: caze.info, results: Outcome.Success([])});
});

from Runner.runCase. It will still report on suites where all test cases have been filtered out, but I guess there is no better way.

[kevinresol]

Right, tink_testrunner doesn't try to catch exceptions because it supports async tests (and it can't catch exceptions from async calls anyway). It is users' responsibility to catch them and produce an Error.

I think this will work:

public function testError() {
 	return new tink.core.Error("failure");
}

[kevinresol]

As for @:exclude, the runner filters out the excluded / non-included cases and doesn't even tries to do anything with them. Maybe we need to change that.

[AlexHaxe]

I think this will work:

public function testError() {
 	return new tink.core.Error("failure");
}

thanks, that helped fixing error handling

[Gama11]

tink_unittest support is now available in 1.2.0. :)