Result XML inconsistent on '@dataProvider returns empty array' or 'is not existing'

[daniela-waranie]

If a data provider function returns an emty array the results XML contains some XML elements about the skipped test (the test method that uses this data provider).

When defining a test method and using a @dataProvider annotation, but pointing to a non existing data provider function this should be create the same XML about the skipped test.

Because of the missing XML about the skipped test (second example above) i can not notice the skipped test in my IDE (netbeans).

[daniela-waranie]

To make it clear: i can not notice the skipped test in the "Test Results" window of netbeans. In the command line output (not the XML) i can see the skipped test. Commandline output and xml should be semantic equal.

[whatthejeff]

The actual issue is that PHPUnit_Util_Log_JUnit ignores PHPUnit_Framework_Warning errors (which is what you get when you specify a @dataProvider that doesn't exist).

[whatthejeff]

For what it's worth, I fixed this in a local branch but it didn't solve the problem with netbeans.

Currently the JUnit result printer will not print the same count for test cases and failures or the same errors as the other result printers if any warnings are encountered. Even after fixing the JUnit result printer to match the other result printers, neatbeans still prints misleading results.

It seems that neatbeans is only looking at the individual test cases for failures and not the overall test suite results. This is problematic as some warnings refer to errors that occur outside of a specific test case. It's possible to create a specific class of warning that relates to warnings that are encountered while trying to execute a specific test case and then handle those in a way that netbeans can recognize but that doesn't solve the overall problem. I think this is something that will have to be fixed in neatbeans as well.

[daniela-waranie]

If your data provider returns an empty array your xml looks like this:
<?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="Webapp_Model_PhpUnitTest" file="S:\myproject\test\myproject\application\models\webapp\models\phpunitTest.php" fullPackage="Webapp.Model" tests="1" assertions="1" failures="0" errors="0" time="0.000676"> <testcase name="testSomething" class="Webapp_Model_PhpUnitTest" file="S:\myproject\test\myproject\application\models\webapp\models\phpunitTest.php" line="9" assertions="1" time="0.000676"/> <testsuite name="Webapp_Model_PhpUnitTest::testMyThing" tests="0" assertions="0" failures="0" errors="0" time="0.000000"/> </testsuite> </testsuites>

Netbeans shows test suite "Webapp_Model_PhpUnitTest::testMyThing" as skipped. Sorry i can not add a snapshot here.

When data provider method is missing the xml for the test suite is missing, too. Which makes netbeans not to show an entry for the skipped test (Webapp_Model_PhpUnitTest::testMyThing).

The attributes of the xml element (testsuite)) should be "tests="0" assertions="0" failures="0" errors="0" time="0.000000"" for both:
a) emty array
b) missing data provider method

On command line both should result in "FAILURES! Tests: 2, Assertions: 1, Failures: 1". An empty array makes no sense and is a failure.

[daniela-waranie]

Netbeans does not consider the command line output. It considers the xml only. Did you add the missing XML element? If your xml element is equal to <testsuite name="Webapp_Model_PhpUnitTest::testMyThing" tests="0" assertions="0" failures="0" errors="0" time="0.000000"/> then netbeans will recognize it as a skipped test suite.

[whatthejeff]

Okay, I fixed this issue here: http://github.com/whatthejeff/phpunit/tree/topic_13. I made the following changes:

• Data providers that return an empty array now generate a warning that says, 'No tests found in suite.' This is more consistent with how normal test suites behave and will generate a warning in CLI or an empty <testsuite> element in the JUNIT format.
• All other errors that occur while setting up a data provider test suite now generate an empty <testsuite> element in the JUNIT format.

I still think that the JUNIT result printer could be more consistent. Judging from the netbeans source code, netbeans generates the "skipped" message because it knows that an empty test suite normally triggers a warning but it has no access to the actual warning in the JUNIT results. It seems like it would make more sense (and would be more consistent with the actual JUNIT DTD) if warnings were reported inside of the <tesetsuite> elements. For example:

<testsuite name="ATest" tests="0" assertions="0" failures="1" errors="0" time="0.000000">
  <failure type="PHPUnit_Framework_Warning">ATest::testA
No tests found in class "ATest".
  </failure>
</testsuite>

This would make the JUNIT results printer more consistent with the other result printers and it would allow applications like neatbeans the possibility of handling warnings more gracefully.

[daniela-waranie]

Would be nice if sebastian will merge this soon.

[daniela-waranie]

I hope to see this soon in 3.5.0RCx

[daniela-waranie]

Hi whatthejeff,

i can not get the result by using your code. I did a checkout on http://github.com/whatthejeff/phpunit.git and copy the "PHPUnit" folder to the PEAR folder of my XAMPP. But i did not get the <testsuite> element in the result xml (when i used an invalid @dataProvider method name when using the --log-junit). Issue not fixed?

[whatthejeff]

Make sure you checkout the topic_13 branch as this bug is fixed in that branch and not the master branch. Something along the lines of:

git checkout topic_13

[daniela-waranie]

Sorry for my incompetence (this is my first time in using git). When i use http://github.com/whatthejeff/phpunit/tree/topic_13 as URL to clone repo via TortoiseGit (Windows) i get the error: fatal: http://github.com/whatthejeff/phpunit/tree/topic_13/info/refs not found: did you run git update-server-info on the server?

[whatthejeff]

Just use the download source button at the top of this page: http://github.com/whatthejeff/phpunit/tree/topic_13.

[sebastianbergmann]

Closed by 5a83f4a.