[SR-9133] JUnit output format doesn't work if --parallel not selected

[alblue]

Previous ID	SR-9133
Radar	None
Original Reporter	@alblue
Type	Bug

Additional Detail from JIRA

Votes	0
Component/s	Package Manager
Labels	Bug, Screened
Assignee	anayini (JIRA)
Priority	Medium

md5: 7a7f61fc246593e1da18eec9acf288a5

Issue Description:

The JUnit output report should be printed regardless of the setting of --parallel.

Running:
swift test --xunit-output=/tmp/results.xml

doesn't generate a file at /tmp/results.xml.

Running:
swift test --xunit-output=/tmp/results.xml --parallel --num-workers=1

does generate a file at /tmp/results.xml

[swift-ci]

Comment by Arjun Nayini (JIRA)

@aciidb0mb3r - I picked up this ticket and started on it. I found that the `TestRunner` and `ParallelTestRunner` are actually separate `final class` types that aren't tied together by anything. I was thinking I could get this done but would want to refactor those types to adhere to a single protocol (refactor them to look alike) and then add in xunit output for the serial one. Currently it looks like the xunit generator class is just explicitly tied to the parallel test runner.

Does this seem like a reasonable approach here? Figured I would ping you since you have the latest commits in that area of the code.

[swift-ci]

Comment by Arjun Nayini (JIRA)

Actually I need to think about this a bit more. Seems like perhaps a SerialTestRunner type might make things a bit more clear with both ParallelTestRunner and SerialTestRunner encapsulating some set of TestRunners.

[Samasaur1]

Any news on this?

[jd-sandk]

It would be nice if this feature/option (--xunit-output) worked as expected, can something please be done to fix this issue.

[michalszelagsonos]

Bump for this issue. We're about to enter 2024, would be great to get something as foundational as this fixed. Note, this kind of report is very important for CI and observability integrations.

[grynspan]

Tracked internally as rdar://129450002. (I'm pretty sure it's tracked with an older radar number too, but I can't find it right now.)

[jakepetroules]

Also, why is the option labeled xunit? It looks like this format is actually based on junit (from Java), while xunit appears to be something that's part of the .NET ecosystem.

[Samasaur1]

Also, why is the option labeled xunit? It looks like this format is actually based on junit (from Java), while xunit appears to be something that's part of the .NET ecosystem.

There is an xUnit.net project that is for the .NET ecosystem, but xunit has also been used to refer to a family of testing frameworks (the idea being that the x refers to the language). See Wikipedia.

[grynspan]

The naming of the flag predates me joining the team. For what it's worth, swift-testing does call it JUnit internally. 🤷🏻‍♂️

[michalszelagsonos]

Bumping this issue. We're approaching 2025, can this fix be prioritized?

[plemarquand]

Are we sure this is still an issue? I'm using a nightly toolchain and I just created a simple test project with two XCTest suites and ran:

swift test --xunit-output=./results.xml:

<?xml version="1.0" encoding="UTF-8"?>

<testsuites>
<testsuite name="TestResults" errors="0" tests="2" failures="0" time="0.095531209">
<testcase classname="AAATests.MySuite" name="testExample" time="0.046890917">
</testcase>
<testcase classname="AAATests.MySuite2" name="testExample2" time="0.048640292">
</testcase>
</testsuite>
</testsuites

And then: swift test --xunit-output=./results.xml --parallel --num-workers=1

<?xml version="1.0" encoding="UTF-8"?>

<testsuites>
<testsuite name="TestResults" errors="0" tests="2" failures="0" time="0.09669525100000001">
<testcase classname="AAATests.MySuite" name="testExample" time="0.049329584">
</testcase>
<testcase classname="AAATests.MySuite2" name="testExample2" time="0.047365667">
</testcase>
</testsuite>
</testsuites>

[grynspan]

I am suspicious of your results as Swift Package Manager doesn't know how to scrape results from XCTest when it runs serially. 🤨

[plemarquand]

Yeah looking at the code in SwiftPM I noticed that, but results.xml seems to be being generated by the swiftpm-testing-helper in the default non parallel case, not SwiftPM. You can see yourself by running:

DYLD_FRAMEWORK_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks DYLD_LIBRARY_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/libexec/swift/pm/swiftpm-testing-helper --test-bundle-path /Users/plemarquand/work/testapps/AAA/.build/arm64-apple-macosx/debug/AAAPackageTests.xctest/Contents/MacOS/AAAPackageTests --xunit-output=./results.xml

[grynspan]

The helper tool only knows how to invoke Swift Testing. What you're seeing is the XML output from Swift Testing (which does know how to generate it when tests run serially.)

[plemarquand]

Sigh, I had a typo in my command to remove old .xml, I am seeing the behaviour described in the initial report now.

[plemarquand]

Seems like this could be implemented with #6861, though that feature is hidden behind an experimental flag, and seems limited to only Darwin today.

[grynspan]

IIRC there were a number of problems with that PR but it's been ages since I looked at it.

And yes, the XCTestObservation protocol is only implemented in XCTest.framework, not corelibs-xctest. That doesn't mean it can't be implemented (approximately, ignoring ObjC minutiae) but it's always a question of engineering resources.

[grynspan]

Note that XCTestObservation has a fair bit of conceptual overlap with the JSON stream in Swift Testing and, I think ideally, we'd want to move XCTest toward Swift Testing rather than reimplementing yet more features in corelibs-xctest. (But that's a question for the upcoming Testing Workgroup Whatchamacallit.)