Request for ability to order/filter test suites and/or tests without changes to underlying test code.

[padraic]

Q	A
PHPUnit version	7.0.0 (version designated to restrict Listeners)
PHP version	7.x.x
Installation Method	Composer / PHAR

In light of #2477 which would remove the current ability to (sneakily, I know!) manipulate test suites within Listeners, this is an issue to request some alternative means of injecting a class with write access. Currently this can be provided within test classes, however there may be some advantage to doing this indirectly as Listeners currently allow.

To explain my own specific needs, a few of us are bringing Humbug back to life, as a mutation testing framework. This involves generating small file changes, and running PHPUnit. We run phpunit for every single change...potentially hundreds of times. To improve performance, we take a number of basic steps described here: https://github.com/humbug/humbug#performance which can be summarised as:

1. Only run tests on covered lines.
2. Eliminate tests not applicable to a line.
3. Order tests fastest-first to hopefully reach a failure quickly.
4. Stop on first failure/error/exception (serves no purpose to continue after one).

Technically, we only order (or filter) test suites - I'll need to double check if that was a limitation of using Listeners, or of PHPUnit in general, or a limitation in our analysis of code coverage data. We do not order individual tests within a parent suite (unless the children are themselves suites) which would actually be the preferred approach.

The performance benefits are central to the framework being useful, as we are not running all tests all the time hundreds of times.

TL;DR - One way of summarising this is getting an array of tests (which would be multidimensional, grouping tests by suite), manipulating that array, injecting a revised array, all while assuming execution order is simply determined by array index. It could perhaps be done by exporting/importing a name map where avoiding any deeper write level interference with actual objects is preferable.

[epdenouden]

@padraic working on something that might be of interest to you, too: https://github.com/epdenouden/phpunit/tree/breakfast is a change for PHPUnit that will allow tests to run tests in defects-first mode. It is a followup of the #3092 test reordering feature.

[mcfedr]

With the upcoming removal of TestListener it wont be possible to do the filtering that I am currently using startTestSuite to do, will there be a replacement for this? I really want to inject a RecursiveFilterIterator into the filters that are called.

Using this at the moment as a way of running tests in parallel in CI environement - so on one machine we do tests 1-10, second machine does 10-20 etc.

[epdenouden]

@mcfedr thanks for bringing this issue to my attention by commenting. The TestListener rewrite needs more discussion/experimentation. I've also been looking at inserting a filter to refactor the test reordering.

In a few days there's a PHPUnit hackathon, I'll be sure to bring it up

[mcfedr]

If its interesting, this, https://github.com/wizaplace/phpunit-slicer, is the code I've been using up to v8, it doesnt work like this because of the final in TestRunner, so I've changed it to use a TestListener for the moment.