Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attribute #[WithoutErrorHandler] to disable PHPUnit's error handler for a test method #5428

Closed
jrfnl opened this issue Jun 26, 2023 · 4 comments · Fixed by #5430
Closed

Attribute #[WithoutErrorHandler] to disable PHPUnit's error handler for a test method #5428

jrfnl opened this issue Jun 26, 2023 · 4 comments · Fixed by #5430
Assignees
Labels
feature/test-runner CLI test runner type/enhancement A new idea that should be implemented
Milestone

Comments

@jrfnl
Copy link
Contributor

jrfnl commented Jun 26, 2023

Q A
PHPUnit version 10*
PHP version 8.2 (though irrelevant)
Installation Method Composer / PHAR (either)

Summary

Code which relies on retrieving the last error message and acting on it can no longer be tested as error_get_last() will always return null.

This can also lead to additional PHP notices being reported by PHPUnit for errors which would never occur in a real-life situation.

Current behavior

Tests which are testing code which relies on error_get_last() are failing on PHPUnit 10 with notices along these lines:

There were 2 failures:

1) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodA
Failed asserting that exception message '' contains 'Failed to open stream'.

2) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodB
Failed asserting that null is identical to 'Triggering'.

path/to/bug-report-reproduction-scenarios/tests/FooTest.php:22

--

2 tests triggered 2 PHP warnings:

1) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodA
Trying to access array offset on value of type null
path/to/bug-report-reproduction-scenarios/src/Foo.php:17

path/to/bug-report-reproduction-scenarios/tests/FooTest.php:12

2) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodB
Trying to access array offset on value of type null
path/to/bug-report-reproduction-scenarios/tests/FooTest.php:22

path/to/bug-report-reproduction-scenarios/tests/FooTest.php:21

--

1 test triggered 1 PHP deprecation:

1) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodA
Exception::__construct(): Passing null to parameter #1 ($message) of type string is deprecated
path/to/bug-report-reproduction-scenarios/src/Foo.php:17

path/to/bug-report-reproduction-scenarios/tests/FooTest.php:12

How to reproduce

Reproduction scenario available at: jrfnl/bug-report-reproduction-scenarios@b3d9366

GH actions build for the reproduction branch demonstrating the issue: https://github.com/jrfnl/bug-report-reproduction-scenarios/actions/runs/5378708821

Expected behavior

The error_get_last() will return the correct last error message so the code under test runs without problems and can be tested like before.

@jrfnl jrfnl added type/bug Something is broken version/10 Something affects PHPUnit 10 labels Jun 26, 2023
jrfnl added a commit to WordPress/Requests that referenced this issue Jun 26, 2023
The `error_get_last()` function can return `null` if no error occurred, in which case, the `throw new Exception()` statement will run into two new errors:
* `Trying to access array offset on value of type null` for accessing `$error['message']`.
* ... which then leads to a `Exception::__construct(): Passing null to parameter #1 ($message) of type string is deprecated`.

This commit adds some defensive coding to handle the hypothetical situation that `error_get_last()` would return `null` when a file could not be opened.

Note: this is actually a bug in PHPUnit 10 which breaks `error_get_last()`.
We should be able to remove the extra defensive coding once the upstream bug has been fixed.

Upstream bug report: sebastianbergmann/phpunit#5428
jrfnl added a commit to WordPress/Requests that referenced this issue Jun 26, 2023
The `error_get_last()` function can return `null` if no error occurred, in which case, the `throw new Exception()` statement will run into two new errors:
* `Trying to access array offset on value of type null` for accessing `$error['message']`.
* ... which then leads to a `Exception::__construct(): Passing null to parameter #1 ($message) of type string is deprecated`.

This commit adds some defensive coding to handle the hypothetical situation that `error_get_last()` would return `null` when a file could not be opened.

Note: this is actually a bug in PHPUnit 10 which breaks `error_get_last()`.
We should be able to remove the extra defensive coding once the upstream bug has been fixed.

Upstream bug report: sebastianbergmann/phpunit#5428
@sebastianbergmann
Copy link
Owner

I understand that this is a problem and I am sorry that it exists. However, supporting "code which relies on retrieving the last error message and acting on it" is a low priority for me.

I will, of course, consider a pull request that aims to solve this problem (without negative impact for PHPUnit's functionality that is based on handling E_* errors emitted by PHP). But I will not investigate this issue myself.

@sebastianbergmann sebastianbergmann added the feature/test-runner CLI test runner label Jun 26, 2023
@sebastianbergmann
Copy link
Owner

PHPUnit's error handler returns true after it handled an E_* error emitted by PHP. This seems to be the reason why error_get_last() does not work in your tests. Returning false, however, breaks PHPUnit's intended behaviour.

Without patch

PHPUnit 10.2.2 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.7
Configuration: /tmp/bug-report-reproduction-scenarios/phpunit.xml.dist

FF                                                                  2 / 2 (100%)

Time: 00:00.013, Memory: 4.00 MB

There were 2 failures:

1) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodA
Failed asserting that exception message '' contains 'Failed to open stream'.

2) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodB
Failed asserting that null is identical to 'Triggering'.

/tmp/bug-report-reproduction-scenarios/tests/FooTest.php:22

--

2 tests triggered 2 PHP warnings:

1) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodA
Trying to access array offset on value of type null
/tmp/bug-report-reproduction-scenarios/src/Foo.php:17

/tmp/bug-report-reproduction-scenarios/tests/FooTest.php:12

2) Jrf\PHPUnit10\Scenario\Tests\FooTest::testMethodB
Trying to access array offset on value of type null
/tmp/bug-report-reproduction-scenarios/tests/FooTest.php:22

/tmp/bug-report-reproduction-scenarios/tests/FooTest.php:21

FAILURES!
Tests: 2, Assertions: 3, Failures: 2, Warnings: 2.

Patch

diff --git a/src/Runner/ErrorHandler.php b/src/Runner/ErrorHandler.php
index 517f8472e..725eb0157 100644
--- a/src/Runner/ErrorHandler.php
+++ b/src/Runner/ErrorHandler.php
@@ -130,7 +130,7 @@ public function __invoke(int $errorNumber, string $errorString, string $errorFil
                 return false;
         }
 
-        return true;
+        return false;
     }
 
     public function enable(): void

With patch

./vendor/bin/phpunit                                    
PHPUnit 10.2.2 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.7
Configuration: /tmp/bug-report-reproduction-scenarios/phpunit.xml.dist

..                                                                  2 / 2 (100%)

Time: 00:00.013, Memory: 4.00 MB

OK (2 tests, 3 assertions)

However ...

PHPUnit 10.2.2-86-gf9f0bc62ad by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.7
Configuration: /usr/local/src/phpunit/phpunit.xml

.............................................................   61 / 2908 (  2%)
.............................................................  122 / 2908 (  4%)
.............................................................  183 / 2908 (  6%)
.............................................................  244 / 2908 (  8%)
.............................................................  305 / 2908 ( 10%)
.............................................................  366 / 2908 ( 12%)
.............................................................  427 / 2908 ( 14%)
.............................................................  488 / 2908 ( 16%)
.............................................................  549 / 2908 ( 18%)
.............................................................  610 / 2908 ( 20%)
.............................................................  671 / 2908 ( 23%)
.............................................................  732 / 2908 ( 25%)
.............................................................  793 / 2908 ( 27%)
.............................................................  854 / 2908 ( 29%)
.............................................................  915 / 2908 ( 31%)
.............................................................  976 / 2908 ( 33%)
............................................................. 1037 / 2908 ( 35%)
............................................................. 1098 / 2908 ( 37%)
............................................................. 1159 / 2908 ( 39%)
............................................................. 1220 / 2908 ( 41%)
............................................................. 1281 / 2908 ( 44%)
............................................................. 1342 / 2908 ( 46%)
............................................................. 1403 / 2908 ( 48%)
............................................................. 1464 / 2908 ( 50%)
............................................................. 1525 / 2908 ( 52%)
............................................................. 1586 / 2908 ( 54%)
............................................................. 1647 / 2908 ( 56%)
............................................................. 1708 / 2908 ( 58%)
............................................................. 1769 / 2908 ( 60%)
............................................................. 1830 / 2908 ( 62%)
............................................................. 1891 / 2908 ( 65%)
............................................................. 1952 / 2908 ( 67%)
............................................................. 2013 / 2908 ( 69%)
............................................................. 2074 / 2908 ( 71%)
............................................................. 2135 / 2908 ( 73%)
............................................................. 2196 / 2908 ( 75%)
............................................................. 2257 / 2908 ( 77%)
............................................................. 2318 / 2908 ( 79%)
............................................................. 2379 / 2908 ( 81%)
............................................................. 2440 / 2908 ( 83%)
............................................................. 2501 / 2908 ( 86%)
............................F.F..FFF......................... 2562 / 2908 ( 88%)
............FF...F..F..................SFF................... 2623 / 2908 ( 90%)
............F.............................................F.. 2684 / 2908 ( 92%)
............................................................. 2745 / 2908 ( 94%)
............................................................. 2806 / 2908 ( 96%)
............................................................. 2867 / 2908 ( 98%)
.........................................                     2908 / 2908 (100%)

Time: 01:00.216, Memory: 26.00 MB

There were 13 failures:

1) /usr/local/src/phpunit/tests/end-to-end/cli/fail-on/fail-on-deprecation.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 message
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\TestRunnerStopping\DeprecationTest::testOne)
+Test Printed Unexpected Output
+
+Deprecated: message in /usr/local/src/phpunit/tests/end-to-end/_files/stop-on-fail-on/DeprecationTest.php on line 19
+
 Test Finished (PHPUnit\TestFixture\TestRunnerStopping\DeprecationTest::testOne)
 Test Preparation Started (PHPUnit\TestFixture\TestRunnerStopping\DeprecationTest::testTwo)
 Test Prepared (PHPUnit\TestFixture\TestRunnerStopping\DeprecationTest::testTwo)

/usr/local/src/phpunit/tests/end-to-end/cli/fail-on/fail-on-deprecation.phpt:41
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

2) /usr/local/src/phpunit/tests/end-to-end/cli/fail-on/fail-on-notice.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 message
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\TestRunnerStopping\NoticeTest::testOne)
+Test Printed Unexpected Output
+
+Notice: message in /usr/local/src/phpunit/tests/end-to-end/_files/stop-on-fail-on/NoticeTest.php on line 19
+
 Test Finished (PHPUnit\TestFixture\TestRunnerStopping\NoticeTest::testOne)
 Test Preparation Started (PHPUnit\TestFixture\TestRunnerStopping\NoticeTest::testTwo)
 Test Prepared (PHPUnit\TestFixture\TestRunnerStopping\NoticeTest::testTwo)

/usr/local/src/phpunit/tests/end-to-end/cli/fail-on/fail-on-notice.phpt:41
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

3) /usr/local/src/phpunit/tests/end-to-end/cli/fail-on/fail-on-warning.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 message
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\TestRunnerStopping\WarningTest::testOne)
+Test Printed Unexpected Output
+
+Warning: message in /usr/local/src/phpunit/tests/end-to-end/_files/stop-on-fail-on/WarningTest.php on line 19
+
 Test Finished (PHPUnit\TestFixture\TestRunnerStopping\WarningTest::testOne)
 Test Preparation Started (PHPUnit\TestFixture\TestRunnerStopping\WarningTest::testTwo)
 Test Prepared (PHPUnit\TestFixture\TestRunnerStopping\WarningTest::testTwo)

/usr/local/src/phpunit/tests/end-to-end/cli/fail-on/fail-on-warning.phpt:41
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

4) /usr/local/src/phpunit/tests/end-to-end/cli/filter-error-handler/filter-for-error-handler-events-disabled.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 Runtime:       PHP 8.2.7
 Configuration: /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/filter-disabled.xml
 
+
+Deprecated: deprecation in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php on line 21
+
+Notice: notice in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php on line 22
+
+Warning: warning in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php on line 23
+
+Deprecated: deprecation in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php on line 8
+
+Notice: notice in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php on line 9
+
+Warning: warning in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php on line 10
 W                                                                   1 / 1 (100%)
 
-Time: %s, Memory: %s
+Time: 00:00.006, Memory: 4.00 MB
 
 1 test triggered 2 warnings:
 
 1) PHPUnit\TestFixture\FilterErrorHandler\SourceClassTest::testSomething
 * warning
-  %s/src/SourceClass.php:23
+  /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php:23
 
 * warning
-  %s/vendor/VendorClass.php:10
+  /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php:10
 
-%s/tests/SourceClassTest.php:16
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/tests/SourceClassTest.php:16
 
 --
 
@@ @@
 
 1) PHPUnit\TestFixture\FilterErrorHandler\SourceClassTest::testSomething
 * notice
-  %s/src/SourceClass.php:22
+  /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php:22
 
 * notice
-  %s/vendor/VendorClass.php:9
+  /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php:9
 
-%s/tests/SourceClassTest.php:16
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/tests/SourceClassTest.php:16
 
 --
 
@@ @@
 
 1) PHPUnit\TestFixture\FilterErrorHandler\SourceClassTest::testSomething
 * deprecation
-  %s/src/SourceClass.php:21
+  /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php:21
 
 * deprecation
-  %s/vendor/VendorClass.php:8
+  /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php:8
 
-%s/tests/SourceClassTest.php:16
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/tests/SourceClassTest.php:16
 
 OK, but there are issues!
 Tests: 1, Assertions: 1, Warnings: 1, Deprecations: 1, Notices: 1.

/usr/local/src/phpunit/tests/end-to-end/cli/filter-error-handler/filter-for-error-handler-events-disabled.phpt:1
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

5) /usr/local/src/phpunit/tests/end-to-end/cli/filter-error-handler/filter-for-error-handler-events-enabled.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 Runtime:       PHP 8.2.7
 Configuration: /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/filter-enabled.xml
 
+
+Deprecated: deprecation in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php on line 21
+
+Notice: notice in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php on line 22
+
+Warning: warning in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php on line 23
+
+Deprecated: deprecation in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php on line 8
+
+Notice: notice in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php on line 9
+
+Warning: warning in /usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/vendor/VendorClass.php on line 10
 W                                                                   1 / 1 (100%)
 
-Time: %s, Memory: %s
+Time: 00:00.006, Memory: 4.00 MB
 
 1 test triggered 1 warning:
 
 1) PHPUnit\TestFixture\FilterErrorHandler\SourceClassTest::testSomething
 warning
-%s/src/SourceClass.php:23
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php:23
 
-%s/tests/SourceClassTest.php:16
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/tests/SourceClassTest.php:16
 
 --
 
@@ @@
 
 1) PHPUnit\TestFixture\FilterErrorHandler\SourceClassTest::testSomething
 notice
-%s/src/SourceClass.php:22
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php:22
 
-%s/tests/SourceClassTest.php:16
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/tests/SourceClassTest.php:16
 
 --
 
@@ @@
 
 1) PHPUnit\TestFixture\FilterErrorHandler\SourceClassTest::testSomething
 deprecation
-%s/src/SourceClass.php:21
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/src/SourceClass.php:21
 
-%s/tests/SourceClassTest.php:16
+/usr/local/src/phpunit/tests/end-to-end/_files/filter-error-handler/tests/SourceClassTest.php:16
 
 OK, but there are issues!
 Tests: 1, Assertions: 1, Warnings: 1, Deprecations: 1, Notices: 1.

/usr/local/src/phpunit/tests/end-to-end/cli/filter-error-handler/filter-for-error-handler-events-enabled.phpt:1
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

6) /usr/local/src/phpunit/tests/end-to-end/cli/stop-on/stop-on-defect-for-warning.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 message
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\TestRunnerStopping\WarningTest::testOne)
+Test Printed Unexpected Output
+
+Warning: message in /usr/local/src/phpunit/tests/end-to-end/_files/stop-on-fail-on/WarningTest.php on line 19
+
 Test Finished (PHPUnit\TestFixture\TestRunnerStopping\WarningTest::testOne)
 Test Runner Execution Aborted
 Test Suite Finished (PHPUnit\TestFixture\TestRunnerStopping\WarningTest, 2 tests)

/usr/local/src/phpunit/tests/end-to-end/cli/stop-on/stop-on-defect-for-warning.phpt:41
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

7) /usr/local/src/phpunit/tests/end-to-end/cli/stop-on/stop-on-deprecation.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 message
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\TestRunnerStopping\DeprecationTest::testOne)
+Test Printed Unexpected Output
+
+Deprecated: message in /usr/local/src/phpunit/tests/end-to-end/_files/stop-on-fail-on/DeprecationTest.php on line 19
+
 Test Finished (PHPUnit\TestFixture\TestRunnerStopping\DeprecationTest::testOne)
 Test Runner Execution Aborted
 Test Suite Finished (PHPUnit\TestFixture\TestRunnerStopping\DeprecationTest, 2 tests)

/usr/local/src/phpunit/tests/end-to-end/cli/stop-on/stop-on-deprecation.phpt:41
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

8) /usr/local/src/phpunit/tests/end-to-end/cli/stop-on/stop-on-notice.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 message
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\TestRunnerStopping\NoticeTest::testOne)
+Test Printed Unexpected Output
+
+Notice: message in /usr/local/src/phpunit/tests/end-to-end/_files/stop-on-fail-on/NoticeTest.php on line 19
+
 Test Finished (PHPUnit\TestFixture\TestRunnerStopping\NoticeTest::testOne)
 Test Runner Execution Aborted
 Test Suite Finished (PHPUnit\TestFixture\TestRunnerStopping\NoticeTest, 2 tests)

/usr/local/src/phpunit/tests/end-to-end/cli/stop-on/stop-on-notice.phpt:41
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

9) /usr/local/src/phpunit/tests/end-to-end/cli/stop-on/stop-on-warning.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 message
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\TestRunnerStopping\WarningTest::testOne)
+Test Printed Unexpected Output
+
+Warning: message in /usr/local/src/phpunit/tests/end-to-end/_files/stop-on-fail-on/WarningTest.php on line 19
+
 Test Finished (PHPUnit\TestFixture\TestRunnerStopping\WarningTest::testOne)
 Test Runner Execution Aborted
 Test Suite Finished (PHPUnit\TestFixture\TestRunnerStopping\WarningTest, 2 tests)

/usr/local/src/phpunit/tests/end-to-end/cli/stop-on/stop-on-warning.phpt:41
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

10) /usr/local/src/phpunit/tests/end-to-end/event/php-notice.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 Only variables should be assigned by reference
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\Event\PhpNoticeTest::testPhpNotice)
+Test Printed Unexpected Output
+
+Notice: Only variables should be assigned by reference in /usr/local/src/phpunit/tests/end-to-end/event/_files/PhpNoticeTest.php on line 22
+
 Test Finished (PHPUnit\TestFixture\Event\PhpNoticeTest::testPhpNotice)
 Test Suite Finished (PHPUnit\TestFixture\Event\PhpNoticeTest, 1 test)
 Test Runner Execution Finished
 Test Runner Finished
 PHPUnit Finished (Shell Exit Code: 1)

/usr/local/src/phpunit/tests/end-to-end/event/php-notice.phpt:43
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

11) /usr/local/src/phpunit/tests/end-to-end/event/php-warning.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 Undefined variable $b
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\Event\PhpWarningTest::testPhpWarning)
+Test Printed Unexpected Output
+
+Warning: Undefined variable $b in /usr/local/src/phpunit/tests/end-to-end/event/_files/PhpWarningTest.php on line 18
+
 Test Finished (PHPUnit\TestFixture\Event\PhpWarningTest::testPhpWarning)
 Test Suite Finished (PHPUnit\TestFixture\Event\PhpWarningTest, 1 test)
 Test Runner Execution Finished
 Test Runner Finished
 PHPUnit Finished (Shell Exit Code: 1)

/usr/local/src/phpunit/tests/end-to-end/event/php-warning.phpt:43
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

12) /usr/local/src/phpunit/tests/end-to-end/event/user-deprecated.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 message
 Assertion Succeeded (Constraint: is true, Value: true)
 Test Passed (PHPUnit\TestFixture\Event\DeprecatedFeatureTest::testDeprecatedFeature)
+Test Printed Unexpected Output
+
+Deprecated: message in /usr/local/src/phpunit/tests/end-to-end/event/_files/DeprecatedFeatureTest.php on line 20
+
 Test Finished (PHPUnit\TestFixture\Event\DeprecatedFeatureTest::testDeprecatedFeature)
 Test Suite Finished (PHPUnit\TestFixture\Event\DeprecatedFeatureTest, 1 test)
 Test Runner Execution Finished
 Test Runner Finished
 PHPUnit Finished (Shell Exit Code: 0)

/usr/local/src/phpunit/tests/end-to-end/event/user-deprecated.phpt:42
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

13) /usr/local/src/phpunit/tests/end-to-end/generic/outcome-and-issues.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 
 Runtime:       PHP 8.2.7
 
-.RDNWFFFEEEDNWDNW                                                 17 / 17 (100%)
+.R
+Deprecated: deprecation message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 32
+D
+Notice: notice message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 39
+N
+Warning: warning message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 46
+W
+Deprecated: deprecation message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 53
+F
+Notice: notice message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 60
+F
+Warning: warning message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 67
+F
+Deprecated: deprecation message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 74
+E
+Notice: notice message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 81
+E
+Warning: warning message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 88
+E
+Deprecated: deprecation message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 95
+D
+Notice: notice message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 102
+N
+Warning: warning message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 109
+W
+Deprecated: deprecation message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 116
+D
+Notice: notice message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 123
+N
+Warning: warning message in /usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php on line 130
+W                                                 17 / 17 (100%)
 
-Time: %s, Memory: %s
+Time: 00:00.018, Memory: 4.00 MB
 
 There were 3 errors:
 
@@ @@
 1) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithDeprecation
 Exception: exception message
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:76
 
 2) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithNotice
 Exception: exception message
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:83
 
 3) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithWarning
 Exception: exception message
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:90
 
 --
 
@@ @@
 1) PHPUnit\TestFixture\OutcomesAndIssuesTest::testFailWithDeprecation
 Failed asserting that false is true.
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:55
 
 2) PHPUnit\TestFixture\OutcomesAndIssuesTest::testFailWithNotice
 Failed asserting that false is true.
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:62
 
 3) PHPUnit\TestFixture\OutcomesAndIssuesTest::testFailWithWarning
 Failed asserting that false is true.
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:69
 
 --
 
@@ @@
 1) PHPUnit\TestFixture\OutcomesAndIssuesTest::testSuccessWithRisky
 This test did not perform any assertions
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:26
 
 2) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithDeprecation
 This test did not perform any assertions
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:72
 
 3) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithNotice
 This test did not perform any assertions
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:79
 
 4) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithWarning
 This test did not perform any assertions
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:86
 
 --
 
@@ @@
 1) PHPUnit\TestFixture\OutcomesAndIssuesTest::testIncompleteWithDeprecation
 incomplete message
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:97
 
 2) PHPUnit\TestFixture\OutcomesAndIssuesTest::testIncompleteWithNotice
 incomplete message
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:104
 
 3) PHPUnit\TestFixture\OutcomesAndIssuesTest::testIncompleteWithWarning
 incomplete message
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:111
 
 --
 
@@ @@
 
 1) PHPUnit\TestFixture\OutcomesAndIssuesTest::testSuccessWithWarning
 warning message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:46
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:44
 
 2) PHPUnit\TestFixture\OutcomesAndIssuesTest::testFailWithWarning
 warning message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:67
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:65
 
 3) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithWarning
 warning message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:88
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:86
 
 4) PHPUnit\TestFixture\OutcomesAndIssuesTest::testIncompleteWithWarning
 warning message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:109
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:107
 
 5) PHPUnit\TestFixture\OutcomesAndIssuesTest::testSkippedWithWarning
 warning message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:130
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:128
 
 --
 
@@ @@
 
 1) PHPUnit\TestFixture\OutcomesAndIssuesTest::testSuccessWithNotice
 notice message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:39
 
-/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:67
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:37
 
 2) PHPUnit\TestFixture\OutcomesAndIssuesTest::testFailWithNotice
 notice message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:60
 
-/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:88
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:58
 
 3) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithNotice
 notice message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:81
 
-/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:109
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:79
 
 4) PHPUnit\TestFixture\OutcomesAndIssuesTest::testIncompleteWithNotice
 notice message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:102
 
-/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:130
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:100
 
 5) PHPUnit\TestFixture\OutcomesAndIssuesTest::testSkippedWithNotice
 notice message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:123
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:121
 
 --
 
@@ @@
 
 1) PHPUnit\TestFixture\OutcomesAndIssuesTest::testSuccessWithDeprecation
 deprecation message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:32
 
-/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:60
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:30
 
 2) PHPUnit\TestFixture\OutcomesAndIssuesTest::testFailWithDeprecation
 deprecation message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:53
 
-/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:81
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:51
 
 3) PHPUnit\TestFixture\OutcomesAndIssuesTest::testErrorWithDeprecation
 deprecation message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:74
 
-/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:102
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:72
 
 4) PHPUnit\TestFixture\OutcomesAndIssuesTest::testIncompleteWithDeprecation
 deprecation message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:95
 
-/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:123
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:93
 
 5) PHPUnit\TestFixture\OutcomesAndIssuesTest::testSkippedWithDeprecation
 deprecation message
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:116
 
-%sOutcomesAndIssuesTest.php:%d
+/usr/local/src/phpunit/tests/end-to-end/_files/OutcomesAndIssuesTest.php:114
 
 ERRORS!
 Tests: 17, Assertions: 7, Errors: 3, Failures: 3, Warnings: 5, Deprecations: 5, Notices: 5, Skipped: 3, Incomplete: 3, Risky: 4.

/usr/local/src/phpunit/tests/end-to-end/generic/outcome-and-issues.phpt:22
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/Framework/TestSuite.php:340
/usr/local/src/phpunit/src/TextUI/TestRunner.php:63
/usr/local/src/phpunit/src/TextUI/Application.php:168

FAILURES!
Tests: 2908, Assertions: 7763, Failures: 13, Skipped: 1.

@sebastianbergmann
Copy link
Owner

As I wrote before, I consider testing code that interacts with PHP's error handling (custom error handler, using error_get_last(), etc.) to be a niche use case that is not a priority for me. And it must not get in the way of PHPUnit's functionality.

That being said, I think there could be a "compromise" in the form of an option to disable PHPUnit's error handler. I am imagining a #[WithoutErrorHandler] attribute that can be used on a test method. Would that help you, Juliette?

@jrfnl
Copy link
Contributor Author

jrfnl commented Jun 26, 2023

@sebastianbergmann Thanks for looking into this. I've applied a quick & dirty patch for the tests through which I discovered the issue.

I think an attribute, like you are proposing, would be a cleaner solution and may also help other people running into this issue.

@sebastianbergmann sebastianbergmann added type/enhancement A new idea that should be implemented and removed type/bug Something is broken labels Jun 26, 2023
@sebastianbergmann sebastianbergmann changed the title PHPUnit 10 breaks error_get_last() Option to disable PHPUnit's error handler Jun 26, 2023
@sebastianbergmann sebastianbergmann removed the version/10 Something affects PHPUnit 10 label Jun 26, 2023
@sebastianbergmann sebastianbergmann self-assigned this Jun 26, 2023
@sebastianbergmann sebastianbergmann added this to the PHPUnit 10.3 milestone Jun 26, 2023
@sebastianbergmann sebastianbergmann changed the title Option to disable PHPUnit's error handler Attribute #[WithoutErrorHandler] to disable PHPUnit's error handler for a test method Jun 27, 2023
sebastianbergmann added a commit that referenced this issue Jun 27, 2023
sebastianbergmann added a commit that referenced this issue Jun 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/test-runner CLI test runner type/enhancement A new idea that should be implemented
Projects
None yet
2 participants