Increase information in output.xml

[meatball133]

Description

When I tested using 0.10.0 of swift-testing and swift 6, the output.xml didn't include information supplied to the @Test annotation. I would personally like for that information to be included in the output.xml file.

Expected behavior

Using this code:

import Testing

@testable import SingleThatPasses

extension Tag {
  @Tag static var task_1: Self
  @Tag static var task_2: Self
  @Tag static var task_3: Self
  @Tag static var task_4: Self
}

@Test("cool test", .tags(.task_1)) func Add() {
  #expect(sum(2, 3) == 5, "2+3 should equal 5")
}

I would expect:

<testsuites>
  <testsuite name="TestResults" errors="0" tests="1" failures="0" time="0.002104295">
    <testcase classname="SingleThatPassesTests" name="Add()" time="0.001154588" description="cool test" />
      <tag name="task_1"></tag>
  </testsuite>
</testsuites>

Actual behavior

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="TestResults" errors="0" tests="1" failures="0" time="0.002104295">
    <testcase classname="SingleThatPassesTests" name="Add()" time="0.001154588" />
  </testsuite>
</testsuites>

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[stmontgomery]

I can research this myself when I have time, but @meatball133 do you happen to know whether the JUnit XML format already has a defined format for representing arbitrary custom tags? Or is <tag> an entirely new element you are proposing? If you could provide any links to references about the format that'd be helpful. Again, we can investigate this too but it just might help move more quickly.

In general, we'd prefer to stick to whatever format has been established for representing tags if there is one, and if there is none, we'll have to consider the potential risk of including undocumented XML nodes.

[meatball133]

Both the addition of the "description" tag in testcase and the tag object was something I came up with. The naming I personally do not mind. As far as I can tell, the official junit standard allows for properties under a test case and under that property, which these tags could be.

The result could look like this:

<testsuites>
  <testsuite name="TestResults" errors="0" tests="1" failures="0" time="0.002104295">
    <testcase classname="SingleThatPassesTests" name="Add()" time="0.001154588" />
      <properties>
       <property name="tag" value="task_1"/>
       <property name="description" value="cool test"/>
     </properties>
  </testsuite>
</testsuites>

This is the source I used for this btw: https://github.com/testmoapp/junitxml

[grynspan]

The XML output provided by Swift Testing matches what is provided by XCTest. We aren't currently planning to add additional functionality here as we don't have proper XML output for either library.

[meatball133]

Surely, it matches the XCTest, even surpassing it in that it actually works without giving the parallel flag and without telling you that the flag is required for it to work.

I personally feel like XML deserves more attention as it has been lackluster for an extended period. What I just mentioned should have been seen as a critical bug but has been apparent from at least 5.7 to 6.0 in XCTest (I have not tested earlier versions, but I would guess it has been that way since XML support was added). XML support is critical for building test-monitoring software that can run outside of XCode, but I personally feel like the support is lackluster just because of just XCode, to keep developers needing to use a Mac with XCode.

Improved XML support could drastically improve developer experience on both Windows and Linux. But also for mac developers not using xcode

I understand that you are likely on a release schedule for Swift 6.0, but now, when a new library is built, I don't see why new functionality can't be added. I am more putting this as a feature request for perhaps future releases, not necessarily Swift 6.0.

[grynspan]

Surely, it matches the XCTest, even surpassing it in that it actually works without giving the parallel flag and without telling you that the flag is required for it to work.

I assume you're referring to swiftlang/swift-package-manager#4752 here. You'll note it's assigned to me in GitHub. :)

XML support is critical for building test-monitoring software that can run outside of XCode, but I personally feel like the support is lackluster just because of just XCode, to keep developers needing to use a Mac with XCode.

You are incorrect.

Improved XML support could drastically improve developer experience on both Windows and Linux. But also for mac developers not using xcode

Our focus for tools interoperability in Swift Testing is a JSON-based ABI. I'd encourage you to check that out. I understand that not all tools can speak our particular dialect of JSON, however there are a variety of benefits to using JSON including the ability to stream results as they occur (rather than providing a single XML document at the end of a test run.)

I understand that you are likely on a release schedule for Swift 6.0, but now, when a new library is built, I don't see why new functionality can't be added. I am more putting this as a feature request for perhaps future releases, not necessarily Swift 6.0.

The goal for the Swift 6 release (specifically in re XML output) is feature parity with XCTest only, and the team's energy is focused on getting a great Swift 6 release out the door. Once the dust has settled and we're looking at features and improvements for the next Swift release after 6, we can consider changes like this one. :)

[grynspan]

#549 tracks skipped tests—although XCTest doesn't report them, JUnit defines the XML schema for skipped tests, so I can justify adding them to Swift Testing (no need to invent any new wheels.) Adding support is simple enough that I can sneak it in when nobody's looking @briancroom okays it.

[grynspan]

Tracked internally as rdar://130619614.