Introduce support for programmatic test suites

[donnerpeter]

Overview

JUnit 3 had "static Test suite()" method where you could create suites as you wish: e.g. first add one class, then another one repeated 100 times, then another one, etc. That's quite handy when you have unstable tests that also spoil some global state, and want to increase the probability of them failing.

JUnit5 has dynamic tests with similar possibilities, but it seems that currently it only allows to generate suites of test methods (via lambdas). It would be nice if it'd also be possible to add classes there, maybe even nested dynamic test generators (aka suites).

Related Issues

• Introduce support for declarative test suites #744

[jbduncan]

@sbrannen I wonder if it's worth inviting the Guava team to this discussion. They use TestSuite a lot by the looks of things in guava-testlib to implement dynamic test generators for their collection classes, so they may be able to advise as to which direction this enhancement should go.

[sbrannen]

Interesting proposal!

We're busy with M1 at the moment. So we'll have to dive deeper into this later.

[kotse]

I am wondering what this feature may look like.

In a project I am working on, we use TestSuite a lot to split our slower running tests into multiple groups. Like take all tests, split them in 4, take the first quarter, execute. Take the second quarter, execute, etc... We can run them in parallel easily. As they are not declared, but discovered at runtime, there is no need to maintain anything - we do not care if a test is moved into another group.

So what we imagine is to have something like this:

@Suite
class MySuite {

  static Collection<Class> suite() { ... }

suite() may return some kind of collection, array, or Stream - something we can iterate over. Like the @MethodSource used by @ParameterizedTest.

We can of course have the above wrapped inside some kind of object - like TestSuite as it is in Junit 3. Maybe this will be more intuitive to people that know the old model.

I have implemeted a very simple test engine that looks for the above to discover tests and run them after that using Junit 5 Engine code.

[smoyer64]

@kotse #681 moved the annotations used by the console runner the junit-platform-suite-api module. There's a proposal for how those annotations might be used to declaratively create a suite with quite a bit of discussion in #744. That issue starts with a block of code that won't run today but represents one way those annotations might be used.

[kotse]

@smoyer64 Yes, I followed the discussion there and have even played around with the code from the experimental branch.

The discussion there is about a declarative suite support, but that will not solve the use case I described.