In CI, verify that PHPunit is actually running

[noahtallen]

What?

This PR adds a check to the unit test workflow which makes sure that PHPunit is executing correctly.

Why?

In #50411, we found that all PHPUnit suites on trunk were not running at all. However, this did not fail the test suite, which meant the issue was undetected for more than a week.

How?

In the PHPUnit github workflow, add a grep call which extracts the number of completed tests from a phpunit log file, and makes sure that at least 500 tests are passing. (This is about half of the current test suite, which seems reasonable when we mostly want to make sure we're

There are a few different scenarios for test output:

1. Tests do not run. The output looks like this:

ℹ Starting '/var/www/html/wp-content/plugins/gutenberg/vendor/bin/phpunit -c /var/www/html/wp-content/plugins/gutenberg/phpunit.xml.dist --verbose' on the tests-wordpress container. 

✔ Ran `/var/www/html/wp-content/plugins/gutenberg/vendor/bin/phpunit -c /var/www/html/wp-content/plugins/gutenberg/phpunit.xml.dist --verbose` in 'tests-wordpress'. (in 0s 701ms)

2. Tests fail.

ℹ Starting '/var/www/html/wp-content/plugins/gutenberg/vendor/bin/phpunit -c /var/www/html/wp-content/plugins/gutenberg/phpunit.xml.dist --verbose' on the tests-wordpress container. 

(...a lot of output we don't care about...)

FAILURES!
Tests: 1181, Assertions: 2152, Failures: 3, Warnings: 34.

3. Tests pass with warnings

(...a lot of output we don't care about...)
WARNINGS!
Tests: 1181, Assertions: 2156, Warnings: 34.

4. Tests pass with no issues:

ℹ Starting '/var/www/html/wp-content/plugins/gutenberg/vendor/bin/phpunit -c /var/www/html/wp-content/plugins/gutenberg/phpunit.xml.dist --verbose' on the tests-wordpress container. 

(...a lot of output we don't care about...)

Time: 5.16 seconds, Memory: 58.00 MB

OK (1120 tests, 2156 assertions)
✔ Ran `/var/www/html/wp-content/plugins/gutenberg/vendor/bin/phpunit -c /var/www/html/wp-content/plugins/gutenberg/phpunit.xml.dist --verbose` in 'tests-wordpress'. (in 6s 164ms)

The grep command looks for a match for OK (1120 tests, and then extracts the number from that and checks that it's more than 500. There is no match in the first two conditions, only in the third.

Testing Instructions

I tested this script in a bash script locally, but I'll also introduce revertable commits which cause an issue in CI so we can verify.

• Passing tests: https://github.com/WordPress/gutenberg/actions/runs/4918745631/jobs/8785474793?pr=50442#step:16:24
• Failing tests: https://github.com/WordPress/gutenberg/actions/runs/4919337124/jobs/8786831880#step:14:252
• Phpunit did not run: https://github.com/WordPress/gutenberg/actions/runs/4919492452/jobs/8787184801?pr=50442#step:16:24

Testing Instructions for Keyboard

Screenshots or screencast

[github-actions]

Flaky tests detected in bbbb38f.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/4919659420
📝 Reported issues:

• [Flaky Test] should warn of unsaved changes before loading reset revision #50419 in /test/e2e/specs/site-editor/user-global-styles-revisions.spec.js

[ObliviousHarmony]

This seems to work well.

The only thing I would say is that perhaps it makes sense for it to be included in the same step as the tests themselves. This doesn't seem like it will be a very common occurrence and so having a step feels like it might be superfluous in virtually all cases.

[noahtallen]

This doesn't seem like it will be a very common occurrence and so having a step feels like it might be superfluous in virtually all cases.

The main reason I did it is just so the script didn't have to be duplicated in both tests, though we could put it in a bash script. IMO, having an extra step doesn't really hurt even if it's superfluous :P