TypeError: PHPUnit\Framework\TestSuiteIterator::current(): Return value must be of type PHPUnit\Framework\Test, null returned

[simPod]

Q	A
PHPUnit version	10.5.21
PHP version	8.3.9
Installation Method	Composer

Summary

v10.5.20 works fine. With 10.5.21 getting the error when I run the whole test suite.

https://github.com/webonyx/graphql-php/actions/runs/9533107528/job/26275958530

Current behavior

TypeError: PHPUnit\Framework\TestSuiteIterator::current(): Return value must be of type PHPUnit\Framework\Test, null returned

How to reproduce

checkout webonyx/graphql-php@21bc031

composer install

vendor/bin/phpunit

Expected behavior

No error.

amphp/amp                             2.6.4              A non-blocking concurrency framework for PHP applications.
amphp/byte-stream                     1.8.2              A stream abstraction to make working with non-blocking I/O simple.
amphp/cache                           1.5.1              A promise-aware caching API for Amp.
amphp/dns                             1.2.3              Async DNS resolution for Amp.
amphp/hpack                           3.2.1              HTTP/2 HPack implementation.
amphp/http                            1.7.3              Basic HTTP primitives which can be shared by servers and clients.
amphp/http-server                     2.1.8              A non-blocking HTTP application server for PHP based on Amp.
amphp/parser                          1.1.1              A generator parser to make streaming parsers simple.
amphp/process                         1.1.7              Asynchronous process manager.
amphp/serialization                   1.0.0              Serialization tools for IPC and data storage in PHP.
amphp/socket                          1.2.1              Async socket connection / server tools for Amp.
amphp/sync                            1.4.2              Mutex, Semaphore, and other synchronization tools for Amp.
amphp/windows-registry                0.3.3              Windows Registry Reader.
cash/lrucache                         1.0.0              An efficient memory-based Least Recently Used (LRU) cache
clue/ndjson-react                     1.3.0              Streaming newline-delimited JSON (NDJSON) parser and encoder for ReactPHP.
composer/pcre                         3.1.4              PCRE wrapping library that offers type-safe preg_* replacements.
composer/semver                       3.4.0              Semver library that offers utilities, version constraint parsing and validation.
composer/xdebug-handler               3.0.5              Restarts a process without Xdebug.
daverandom/libdns                     2.1.0              DNS protocol implementation written in pure PHP
dms/phpunit-arraysubset-asserts       dev-master 1aa0d83 This package provides ArraySubset and related asserts once deprecated in PHPUnit 8
doctrine/annotations                  2.0.1              Docblock Annotations Parser
doctrine/lexer                        3.0.1              PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.
ergebnis/composer-normalize           2.43.0             Provides a composer plugin for normalizing composer.json.
ergebnis/json                         1.2.0              Provides a Json value object for representing a valid JSON string.
ergebnis/json-normalizer              4.5.0              Provides generic and vendor-specific normalizers for normalizing JSON documents.
ergebnis/json-pointer                 3.4.0              Provides an abstraction of a JSON pointer.
ergebnis/json-printer                 3.5.0              Provides a JSON printer, allowing for flexible indentation.
ergebnis/json-schema-validator        4.2.0              Provides a JSON schema validator, building on top of justinrainbow/json-schema.
evenement/evenement                   3.0.2              Événement is a very simple event dispatching library for PHP
fidry/cpu-core-counter                1.1.0              Tiny utility to get the number of CPU cores.
fig/http-message-util                 1.1.5              Utility classes and constants for use with PSR-7 (psr/http-message)
friendsofphp/php-cs-fixer             3.58.1             A tool to automatically fix PHP code style
justinrainbow/json-schema             5.2.13             A library to validate a json schema.
kelunik/certificate                   1.1.3              Access certificate details and transform between different formats.
kubawerlos/php-cs-fixer-custom-fixers 3.21.0             A set of custom fixers for PHP CS Fixer
league/uri                            6.8.0              URI manipulation library
league/uri-interfaces                 2.3.0              Common interface for URI representation
league/uri-parser                     1.4.1              userland URI parser RFC 3986 compliant
localheinz/diff                       1.1.1              Fork of sebastian/diff for use with ergebnis/composer-normalize
mll-lab/php-cs-fixer-config           5.7.0              Shared rules for php-cs-fixer
myclabs/deep-copy                     1.12.0             Create deep copies (clones) of your objects
nikic/php-parser                      5.0.2              A PHP parser written in PHP
nyholm/psr7                           1.8.1              A fast PHP7 implementation of PSR-7
phar-io/manifest                      2.0.4              Component for reading phar.io manifest information from a PHP Archive (PHAR)
phar-io/version                       3.2.1              Library for handling version information and constraints
phpbench/container                    2.2.2              Simple, configurable, service container.
phpbench/dom                          0.3.3              DOM wrapper to simplify working with the PHP DOM implementation
phpbench/phpbench                     1.2.15             PHP Benchmarking Framework
phpstan/extension-installer           1.4.1              Composer plugin for automatic installation of PHPStan extensions
phpstan/phpstan                       1.11.4             PHPStan - PHP Static Analysis Tool
phpstan/phpstan-phpunit               1.4.0              PHPUnit extensions and rules for PHPStan
phpstan/phpstan-strict-rules          1.6.0              Extra strict and opinionated rules for PHPStan
phpunit/php-code-coverage             10.1.14            Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator             4.1.0              FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-invoker                   4.0.0              Invoke callables with a timeout
phpunit/php-text-template             3.0.1              Simple template engine.
phpunit/php-timer                     6.0.0              Utility class for timing
phpunit/phpunit                       10.5.20            The PHP Unit Testing framework.
psr/cache                             3.0.0              Common interface for caching libraries
psr/container                         2.0.2              Common Container Interface (PHP FIG PSR-11)
psr/event-dispatcher                  1.0.0              Standard interfaces for event handling.
psr/http-factory                      1.1.0              PSR-17: Common interfaces for PSR-7 HTTP message factories
psr/http-message                      1.1                Common interface for HTTP messages
psr/log                               3.0.0              Common interface for logging libraries
react/cache                           1.2.0              Async, Promise-based cache interface for ReactPHP
react/child-process                   0.6.5              Event-driven library for executing child processes with ReactPHP.
react/dns                             1.13.0             Async DNS resolver for ReactPHP
react/event-loop                      1.5.0              ReactPHP's core reactor event loop that libraries can use for evented I/O.
react/http                            1.10.0             Event-driven, streaming HTTP client and server implementation for ReactPHP
react/promise                         3.2.0              A lightweight implementation of CommonJS Promises/A for PHP
react/socket                          1.15.0             Async, streaming plaintext TCP/IP and secure TLS socket server and client connections for ReactPHP
react/stream                          1.4.0              Event-driven readable and writable streams for non-blocking I/O in ReactPHP
rector/rector                         1.1.0              Instant Upgrade and Automated Refactoring of any PHP code
sebastian/cli-parser                  2.0.1              Library for parsing CLI options
sebastian/code-unit                   2.0.0              Collection of value objects that represent the PHP code units
sebastian/code-unit-reverse-lookup    3.0.0              Looks up which function or method a line of code belongs to
sebastian/comparator                  5.0.1              Provides the functionality to compare PHP values for equality
sebastian/complexity                  3.2.0              Library for calculating the complexity of PHP code units
sebastian/diff                        5.1.1              Diff implementation
sebastian/environment                 6.1.0              Provides functionality to handle HHVM/PHP environments
sebastian/exporter                    5.1.2              Provides the functionality to export PHP variables for visualization
sebastian/global-state                6.0.2              Snapshotting of global state
sebastian/lines-of-code               2.0.2              Library for counting the lines of code in PHP source code
sebastian/object-enumerator           5.0.0              Traverses array structures and object graphs to enumerate all referenced objects
sebastian/object-reflector            3.0.0              Allows reflection of object attributes, including inherited and non-public ones
sebastian/recursion-context           5.0.0              Provides functionality to recursively process PHP variables
sebastian/type                        4.0.0              Collection of value objects that represent the types of the PHP type system
sebastian/version                     4.0.1              Library that helps with managing the version number of Git-hosted PHP projects
seld/jsonlint                         1.10.2             JSON Linter
symfony/console                       7.1.1              Eases the creation of beautiful and testable command line interfaces
symfony/deprecation-contracts         3.5.0              A generic function and convention to trigger deprecation notices
symfony/event-dispatcher              7.1.1              Provides tools that allow your application components to communicate with each other by dispatching events and listening to them
symfony/event-dispatcher-contracts    3.5.0              Generic abstractions related to dispatching event
symfony/filesystem                    7.1.1              Provides basic utilities for the filesystem
symfony/finder                        7.1.1              Finds files and directories via an intuitive fluent interface
symfony/options-resolver              7.1.1              Provides an improved replacement for the array_replace PHP function
symfony/polyfill-ctype                1.29.0             Symfony polyfill for ctype functions
symfony/polyfill-intl-grapheme        1.29.0             Symfony polyfill for intl's grapheme_* functions
symfony/polyfill-intl-normalizer      1.29.0             Symfony polyfill for intl's Normalizer class and related functions
symfony/polyfill-mbstring             1.29.0             Symfony polyfill for the Mbstring extension
symfony/polyfill-php80                1.29.0             Symfony polyfill backporting some PHP 8.0+ features to lower PHP versions
symfony/polyfill-php81                1.29.0             Symfony polyfill backporting some PHP 8.1+ features to lower PHP versions
symfony/process                       7.1.1              Executes commands in sub-processes
symfony/service-contracts             3.5.0              Generic abstractions related to writing services
symfony/stopwatch                     7.1.1              Provides a way to profile code
symfony/string                        7.1.1              Provides an object-oriented API to strings and deals with bytes, UTF-8 code points and grapheme clusters in a unified way
symfony/var-exporter                  7.1.1              Allows exporting any serializable PHP data structure to plain PHP code
thecodingmachine/safe                 2.5.0              PHP core functions that throw exceptions instead of returning FALSE on error
theseer/tokenizer                     1.2.3              A small library for converting tokenized PHP source code into XML and potentially other formats
webmozart/glob                        4.7.0              A PHP implementation of Ant's glob.

[mfn]

Similar breakage with the latest phpunit release

ErrorException: Undefined array key 3 in /project/vendor/phpunit/phpunit/src/Framework/TestSuiteIterator.php:52
Stack trace:
#0 /project/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(255): Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'Undefined array...', '/project/ve...', 52)
#1 /project/vendor/phpunit/phpunit/src/Framework/TestSuiteIterator.php(52): Illuminate\Foundation\Bootstrap\HandleExceptions->Illuminate\Foundation\Bootstrap\{closure}(2, 'Undefined array...', '/project/ve...', 52)
#2 [internal function]: PHPUnit\Framework\TestSuiteIterator->current()
#3 /project/vendor/phpunit/phpunit/src/Framework/TestSuite.php(282): FilterIterator->rewind()
#4 /project/vendor/phpunit/phpunit/src/Framework/TestSuite.php(283): PHPUnit\Framework\TestSuite->count()
#5 /project/vendor/webonyx/graphql-php/src/Error/FormattedError.php(302): PHPUnit\Framework\TestSuite->count()
#6 [internal function]: GraphQL\Error\FormattedError::printVar(Object(PHPUnit\Framework\TestSuite))
#7 /project/vendor/webonyx/graphql-php/src/Error/FormattedError.php(279): array_map(Array, Array)
#8 /project/vendor/webonyx/graphql-php/src/Error/FormattedError.php(218): GraphQL\Error\FormattedError::toSafeTrace(Object(Illuminate\Database\Eloquent\ModelNotFoundException))
#9 /project/vendor/webonyx/graphql-php/src/Error/FormattedError.php(163): GraphQL\Error\FormattedError::addDebugEntries(Array, Object(GraphQL\Error\Error), 3)
#10 /project/app/GraphQL/GraphQL.php(47): GraphQL\Error\FormattedError::createFromException(Object(GraphQL\Error\Error), 3)
#11 /project/vendor/webonyx/graphql-php/src/Error/FormattedError.php(236): Project\GraphQL\GraphQL::formatError(Object(GraphQL\Error\Error))
#12 [internal function]: GraphQL\Error\FormattedError::GraphQL\Error\{closure}(Object(GraphQL\Error\Error))
#13 /project/vendor/rebing/graphql-laravel/src/GraphQL.php(600): array_map(Object(Closure), Array)
#14 /project/vendor/webonyx/graphql-php/src/Executor/ExecutionResult.php(164): Rebing\GraphQL\GraphQL::handleErrors(Array, Object(Closure))
#15 /project/vendor/rebing/graphql-laravel/src/GraphQL.php(139): GraphQL\Executor\ExecutionResult->toArray()
#16 /project/vendor/rebing/graphql-laravel/src/GraphQLController.php(41): Rebing\GraphQL\GraphQL->execute('default', Object(Rebing\GraphQL\Support\OperationParams))
#17 /project/vendor/rebing/graphql-laravel/src/Helpers.php(24): Rebing\GraphQL\GraphQLController->Rebing\GraphQL\{closure}(Object(GraphQL\Server\OperationParams))
#18 /project/vendor/rebing/graphql-laravel/src/GraphQLController.php(37): Rebing\GraphQL\Helpers::applyEach(Object(Closure), Object(GraphQL\Server\OperationParams))
#19 /project/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): Rebing\GraphQL\GraphQLController->query(Object(Illuminate\Http\Request), Object(Laragraph\Utils\RequestParser), Object(Illuminate\Config\Repository), Object(Rebing\GraphQL\GraphQL))
#20 /project/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(43): Illuminate\Routing\Controller->callAction('query', Array)
#21 /project/vendor/laravel/framework/src/Illuminate/Routing/Route.php(259): Illuminate\Routing\ControllerDispatcher->dispatch(Object(Illuminate\Routing\Route), Object(Rebing\GraphQL\GraphQLController), 'query')
#22 /project/vendor/laravel/framework/src/Illuminate/Routing/Route.php(205): Illuminate\Routing\Route->runController()
#23 /project/vendor/laravel/framework/src/Illuminate/Routing/Router.php(806): Illuminate\Routing\Route->run()
#24 /project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\Routing\Router->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
…

Maybe it's not a coincidence that webonyx/graphql-php is involved, too?

I'm seeing similar errors over at https://github.com/rebing/graphql-laravel when I run composer update and run the test suite.

[mfn]

Is #5866 a regression of this?

We had to lock phpunit to 10.5.20 due to this.

[sebastianbergmann]

Thank you for your report.

Please provide a minimal, self-contained, reproducing test case that shows the problem you are reporting.

Without such a minimal, self-contained, reproducing test case I will not be able to investigate this issue.

[mfn]

Sorry meant: possible regression of #5861 ?

Destroy TestCase object after its test was run

Because the error is triggered in \PHPUnit\Framework\TestSuiteIterator::current

[sebastianbergmann]

Yes, #5861 would be the only change in PHPUnit 10.5.21 that could explain this.

[mfn]

I just noticed also a difference:

• when i just run vendor/bin/phpunit
  -> fails as described above
• when I run vendor/bin/phpunit path/to/test/file/which/justfailed/above.php
  -> error not triggered, test is green

Luckily, I can reproduce it when I provide a custom @group foo on the test and run with --group foo filter.

[mfn]

I set a breakpoint on the condition !isset($this->tests[$this->position])

There's no match on that position although there are elements in that array.

It seems there's something in webonyx/graphql-php specifically, which triggers this state.

I think I mostly see this when a test fails, the webonyx/graphql-php catches the error and, as part of the processing of that error, from the screenshot:

• toSafeTrace()
• printVar()

Eventally, printVar() receives a TestSuite object and the generic code therein detects it is of type \Countable and calls count() on it:

        if (\is_object($var)) {
            return 'instance of ' . \get_class($var) . ($var instanceof \Countable ? '(' . \count($var) . ')' : '');
        }

When I remove that count() call -> no problems.

[sebastianbergmann]

I think I mostly see this when a test fails, the webonyx/graphql-php catches the error and, as part of the processing of that error, from the screenshot

If that that is indeed the root cause here then I will close this as "won't fix". An error handler that is not PHPUnit's must not mess with exceptions raised by PHPUnit that indicate test failure.

[mfn]

Hmm, but how is count() on a \Countable messing around with anything? 🤔

[sebastianbergmann]

That is besides the point: the TestSuite object in question, as well as the associated iterator, are internal to PHPUnit's test runner and must not be used from outsite of PHPUnit's test runner. As of #5861, TestCase objects aggregated in TestSuite are delete after their test was run.

[mfn]

Then maybe the \Countable should be removed, as it's not fulfilling the contract?

[simPod]

@mfn you've indeed found the issue, it is not directly related to phpunit. For some reason the count() call on Testcase object triggers the sideeffect via

phpunit/src/Framework/TestSuite.php

Lines 265 to 274 in 1d08c52

	public function count(): int
	{
	$numTests = 0;
	foreach ($this as $test) {
	$numTests += count($test);
	}
	return $numTests;
	}

that invokes FilterIterator and messes with tests indexing.

phpunit/src/Framework/TestSuite.php

Lines 416 to 424 in 1d08c52

	public function getIterator(): Iterator
	{
	$iterator = new TestSuiteIterator($this);
	if ($this->iteratorFilter !== null) {
	$iterator = $this->iteratorFilter->factory($iterator, $this);
	}
	return $iterator;