Empty report when beforeAll is failing

[dubilyer]

Steps to reproduce

1. Create a testClass with parameterized tests only;
2. Create a beforeAll method throwing NPE;
3. Register LegacyXmlReportGeneratingListener.

Because we have:

  void writeXmlReport(TestIdentifier testIdentifier, Writer out) throws XMLStreamException {
      List<TestIdentifier> tests = (List)this.reportData.getTestPlan().getDescendants(testIdentifier).stream().filter(TestIdentifier::isTest).collect(Collectors.toList());
      this.writeXmlReport(testIdentifier, tests, out);
  }

in XmlReportWriter

and all skipped tests are containers (isTest() = false), we have a completely clean xml report with no way to understand, that beforeAll failed and tests were skipped.

Junit version is 5.5.1, platform 1.7.1

[marcphilipp]

Very good point, we should also included failed containers.

[marcphilipp]

@dubilyer Are you sure those tests fail in a @BeforeAll method? If they did, this method would cause the NPE to be reported for each contained test identifier:

junit5/junit-platform-reporting/src/main/java/org/junit/platform/reporting/legacy/xml/XmlReportData.java

Lines 106 to 119 in 17d3b44

	Optional<TestExecutionResult> getResult(TestIdentifier testIdentifier) {
	if (this.finishedTests.containsKey(testIdentifier)) {
	return Optional.of(this.finishedTests.get(testIdentifier));
	}
	Optional<TestIdentifier> parent = this.testPlan.getParent(testIdentifier);
	Optional<TestIdentifier> ancestor = findAncestor(parent, this.finishedTests::containsKey);
	if (ancestor.isPresent()) {
	TestExecutionResult result = this.finishedTests.get(ancestor.get());
	if (result.getStatus() != SUCCESSFUL) {
	return Optional.of(result);
	}
	}
	return Optional.empty();
	}

[dubilyer]

I have a quite complex project, and didn't manage to create a simple reproducer yet, but I'll share my understanding. All tests in a class are parametrized. So we have the following structure (as I remember it :) : [ test-engine] | ---- [.../test-class] | -----[.../test-class/test-method] | ----[.../test-class/test-method/invocation1] ----[.../test-class/test-method/invocation2] -----[.../test-class/test-method] | ----[.../test-class/test-method/invocation1] ----[.../test-class/test-method/invocation2] When beforeAll is failing, we have a failed test-class; and skipped(? not sure, can check) tests method. And it's not even start invocations. So we have an empty list in writeXmlReport method because the stream has been filtered by TestIdentifier::isTest. So nothing is written even if we have something inside XmlReportData. If we have non-parameterized tests, everything works as expected.

[marcphilipp]

I think I figured it out. 🙂

@dubilyer Do you have time to verify my change with your project against the latest 5.7.1-SNAPSHOT version from https://oss.sonatype.org/content/repositories/snapshots/?

[dubilyer]

I will check and update. Thanks a lot!

[dubilyer]

Cannot get a platform dependency with this version.

[marcphilipp]

For platform, it would be 1.7.1-SNAPSHOT. However, we've released 1.7.1/5.7.1 in the meantime.

[dubilyer]

Issue resolved! Thanks a lot. Waiting for release version :)

[sbrannen]

Issue resolved! Thanks a lot. Waiting for release version :)

@dubilyer, it has already been released: https://github.com/junit-team/junit5/releases/tag/r5.7.1