Do not extend final class TestRunner and just copy-and-paste its contents

[spawnia]

This is definitely a hack and a maintenance nightmare - but it works for now.

We can look for a better alternative later, see #7

[spawnia]

It looks like Travis checks are not running. Can you check if the authentication has to be renewed?

[mfn]

I learned about phpunit-slicer from your issue over at sebastianbergmann/phpunit#4121

I've been using this PR (git commit hash 306846a) a few weeks now since moving over to Github Actions in a private repo and it's category "game changer" in terms of cutting down the runtime (~10k tests => went down from initial 2 suites ~15 mins to 6 slices done in ~4-5 mins).

I already tried to come up with something better but ran into all the problems you probably did too and would have ended up copying the file over 🤷‍♀️

I lack the knowledge of phpunit internals (and a bit time) to dig into that. But it seems phpunit is on a pace to seal many of it's openness in order to reduce complexity and too-creative-use on users side. Likely due to all the petty support issues to have to deal with (just a speculation, reading their issues for a while gives me the impression a bit).

[mfn]

Couldn't upgrade to phpunit9 due to version constraint in composer.json

Have not yet found time (will try later) but anyone already knowing if it "just works" or not?

[mfn]

I got it working but I was basically doing the same copypaste galore you already did but with PHPUnit9 code 😄

It becomes pretty clear that any package trying to do this has to be creative because the way phpunit keeps changing currently you can't simply have a single code base support all those versions.

Either you jump through hoops with PHPunit version checks or class_exists stuff for a single code base or the package just keeps dedicated major releases mirroring the respective PHPUnit release.

---

Of course only now I realized that @spawnia was already working on this in https://github.com/mll-lab/phpunit-slicer/commits/phpunit-9 but not PR

---

Not sure who runs the repo, last commits are from @wizacedric and @philippe-vandermoere : are you still interesting in keeping the repo running for newer versions, any thoughts on the matter?

[spawnia]

@mfn quick heads up, the PHPUnit 9 branch only works with version 9.2.5 (potentially lower, have not tried that). 9.2.6 breaks things again. I locked the requirement to make sure https://github.com/mll-lab/phpunit-slicer/commits/phpunit-9 keeps working for now.

[mfn]

@spawnia thanks!

I did not notice anymore as I assimilated the slicer in a private code repo already with 9.2.6 . Call that luck 😅

So we would end up continuously having to copy/paste adapt the two classes…

When I assimilated the code (I need to move fast here) it turns out it's not THAT bad in this bad situation:

• copy over command
• copy over testsuite
• in both apply minimal changes to allow extending it (i.e. g remove final, make methods protected instead of private and remove certain return signatures
• keep the most relevant business logic still inside separate files

So every time the release a new version it's the same over and over again…

I can see myself patching this for a company driven project but not maintaining in an OSS project when other people start to depend on this, as I'll just schedule the upgrades when I've really time for this.

Damn :)

[wizacedric]

@spawnia @mfn Thanks for both your inputs. I don't think we can accept the PR as it is. It would create a hard link between a version of phpunit/phpunit and a version of wizaplace/phpunit-slicer, which would be a problem for most users. We would need to have as many versions of wizaplace/phpunit-slicer as there are versions of phpunit/phpunit...

[wizacedric]

Can https://phpunit.readthedocs.io/en/9.2/extending-phpunit.html#extending-the-testrunner be a way to move forward on this?

[spawnia]

@wizacedric PHPUnit brought us into this mess by locking down their implementation, adding final and private all over the place. At the same time, they did not offer extension mechanisms that are quite flexible enough to achieve what is needed.

Can https://phpunit.readthedocs.io/en/9.2/extending-phpunit.html#extending-the-testrunner be a way to move forward on this?

Maybe using BeforeTestHook and skipping all tests that are not in the current slice? We could probably hack around and (ab-)use static properties to pass the CLI arguments and count the tests.

According to sebastianbergmann/phpunit#4121, PHPUnit would most likely add a solution that allows extensions to control PHPUnit test selection through an XML export/import.

It would create a hard link between a version of phpunit/phpunit and a version of wizaplace/phpunit-slicer, which would be a problem for most users.

Agree it is problematic, still better than no solution at all.

We would need to have as many versions of wizaplace/phpunit-slicer as there are versions of phpunit/phpunit...

Not totally true, we would have to cover more specific ranges, down to the minor version.

Looks like @mfn and me maintain something like this on the side anyways. It is a bit ugly, but not too much of a hassle. Might as well join efforts.

[mfn]

Maybe using BeforeTestHook

I gave it a cursory look so I might be wrong, but the hook doesn't even receive the test lest can return a value or signal in any way how to proceed or not.

Probably would require calling \PHPUnit\Framework\Assert::markTestSkipped if we could even figure out we sliced a particular test, but since the context is basically "string name of test", I doubt that.

The whole extending phpunit chapter, whilst looking nice from above (hooks everywhere), since you basically can't "control" anything isn't really useful for us.

---

I could see a hacky way using global state somewhere, calculating the slice based on counting the test executed (hopefully tests are always run in the same order…) and throw a SkippedTestError.

I've a >10k test suite, if I make 5 slices this means I'll 40k times throw this error to skip the tests… nah, I don't think I'll even attempt this.

---

Want to see a "funny" coincidence?

• me, want to figure out how properly extend via createRunner
  => https://stackoverflow.com/questions/63208917/how-to-extend-phpunit-textui-commandcreaterunner-in-recent-versions-of-phpuni
• "Remove Command::createRunner()"

I can hardly believe (see the timestamps involved) this is a coincidence.

---

Might as well join efforts.

Agree, I made #9 to show the intent I had in mind. It's still copypasta galore, so no real improvement over the status quo.

[mfn]

I've yet to look further but I wonder if the concept of test filters in phpunit can be better (ab)used to "slice", see https://github.com/sebastianbergmann/phpunit/blob/ff047828b43b7ba88300372fb41943ddceb2db03/src/Runner/Filter/Factory.php#L50-L60 🤷‍♀️

[spawnia]

How about we give in and try to get something like sebastianbergmann/phpunit#3387 implemented in PHPUnit?

[mfn]

Yeah I remember it was mentioned in sebastianbergmann/phpunit#4121 (comment) ;)

Btw. this issue links to sebastianbergmann/phpunit#3605 which I wasn't really aware of

And I thought "aha, 'phpunit chunk'" and voila https://www.google.com/search?hl=en&q=phpunit%20chunk => https://github.com/jwage/phpchunkit
But basic installation fails, all dependencies are so outdated I can't install it with L7/PHPUnit9

Not sure if @jwage is still active in this area?

[mfn]

How about we give in and try to get something like sebastianbergmann/phpunit#3387 implemented in PHPUnit?

After having to go through the pain recently again of adapting sources for PHPUnit 9.3, I finally gave it a stab at sebastianbergmann/phpunit#4449

[mfn]

I'm happy (…) to report that the copypaste approach still works with PHPUnit 9.5 :}

I tried #10 but I couldn't get it working, left comments there.

Only now I saw I got feedback in my phpunit PR which I was not ware of, will look at this ASAP too.

[mfn]

Since the signs are not good that the copypaste approach will continue to work as phpunit is changing their internals faster then I change my underwear and sebastianbergmann/phpunit#4449 also doesn't show any movement, I devised a new strategy.

I don't think I'm the first to come up with this but TBH I have not seen this solution somewhere else:

1. create list of the tests in XML format from phpunit:
   phpunit --list-tests-xml all_tests.xml
2. Use a script to "splice" this XML into smaller fragments, based on the idea phpunit-slicer -> phpunit_xml_slicer.php
phpunit_xml_slicer.php all_tests.xml 2/10 > slice_2_10.xml
3. Use yet another script phpunit_xml_class_to_file.php which takes the sliced XML and:
   • builds map of all the test classes
   • using the composer.json matches them for their PSR-4 namespace and uses this to convert them to files
   • replaces any existing testsuites purely with a single suite with all the files from the references test classes from the sliced XML
     phpunit_xml_class_to_file.php composer.json phpunit.xml.dist slice_2_10.xml > phpunit-ci.xml
4. Use the phpunit-ci.xml to run phpunit in CI, which now only contains the <file>s from the sliced XML:
   phpunit --configuration phpunit-ci.xml

This sounds involved, but it's just a few lines added to e.g. Github Action step and presto, you can get almost the same benefit as from phpunit-slicer, except now it's not depending on any PHPUnit internals anymore; pseudo example:

jobs:
  phpunit:
    strategy:
      matrix:
        phpunit-slices: ['1/6', '2/6', '3/6', '4/6', '5/6', '6/6']
    steps:
      # … other steps before
      - name: 'phpunit: export all tests as XML'
        run: vendor/bin/phpunit --list-tests-xml all_tests.xml

      - name: 'phpunit: slice tests ${{ matrix.phpunit-slices }}'
        run: phpunit_xml_slicer.php all_tests.xml ${{ matrix.phpunit-slices }} > slice.xml

      - name: 'phpunit: convert classes to files and inject them back into phpunit XML config'
        run: phpunit_xml_class_to_file.php composer.json phpunit.xml.dist slice.xml > phpunit-ci.xml

      - run: vendor/bin/phpunit --configuration phpunit-ci.xml

Biggest difference

• The approach used by phpunit-slicer can operation on the "test method" level, i.e. very fine grained.
  (Not sure how it handles @dataprovider)
• The approach outlined above operates on the "test class" level, i.e. it's much more coarse

I had concerns that this would create an imbalance of my test suite I'm testing this with (~15k tests) so that some slices would run much longer than others, but turns out in practice the difference was not noticeable. Might me a lucky case for me though, YMMV.

Why?!

I was exploring https://laravel.com/docs/8.x/testing#running-tests-in-parallel the other day and I could not really use phpunit-slicer for this, ran into all sorts of issues and finally took another stab at it.

What about the phpunit PR you mentioned?

sebastianbergmann/phpunit#4449

Not sure when this progresses, but this would be the ideal case for a solution her:

• it would be as fine grained as phpunit-slicer is, i.e. operate on the "individual test" level (including @dataprovider)
• paratest (the underlying tool for Laravels parallel testing) signaled they would support above PR too => Custom test suite loaders are ignored paratestphp/paratest#556 (comment)
• which means I would expect adding it to Laravel would be at least technically possible too

Let's see \o/

---

I spent already way more on these kind of topics I ever thought I would, happy to hear about other perspectives / solutions / approaches 😄

[steamboatid]

@wizacedric PHPUnit brought us into this mess by locking down their implementation, adding final and private all over the place. At the same time, they did not offer extension mechanisms that are quite flexible enough to achieve what is needed.

Can https://phpunit.readthedocs.io/en/9.2/extending-phpunit.html#extending-the-testrunner be a way to move forward on this?

Maybe using BeforeTestHook and skipping all tests that are not in the current slice? We could probably hack around and (ab-)use static properties to pass the CLI arguments and count the tests.

According to sebastianbergmann/phpunit#4121, PHPUnit would most likely add a solution that allows extensions to control PHPUnit test selection through an XML export/import.

It would create a hard link between a version of phpunit/phpunit and a version of wizaplace/phpunit-slicer, which would be a problem for most users.

Agree it is problematic, still better than no solution at all.

We would need to have as many versions of wizaplace/phpunit-slicer as there are versions of phpunit/phpunit...

Not totally true, we would have to cover more specific ranges, down to the minor version.

Looks like @mfn and me maintain something like this on the side anyways. It is a bit ugly, but not too much of a hassle. Might as well join efforts.

@spawnia it's PHP, and its open source.
why not we forking then simply remove the final keyword and replacing the private into protected ?
it seem that's the only way in PHP, because PHP don't support overloading.

I'm facing the same problem as yours. Long run time and in-consistent test results, make me nuts.
It's OK when I run manually by group and filter, but found tons of error and fails when run all at once.

for the time being,
I will patch the code into phpunit directly