Does gotestsum support adding properties to test cases in the junit xml result file ?

[Victor-Chirinian-Vimeo]

Hello,
I was wondering if there was a way to add properties to a test case and be available in the xml result file.

<testcase classname="tests.LoginTests" name="test_case_1" time="650"> <properties> <property name="property1" value="value1"/> </properties>

[dnephin]

Hello!

There is currently no way to do that with gotestsum, but I'd love to learn more about your use case.

What kind of data do you want to associate with the test case? Is it data that gotestsum may already have access to from the test output, or would it come from some other source? If the data is already available to gotestsum, it might be easy to get those properties added to the xml.

How will these properties be consumed? Is there another system that will read these properties?

XSLT is a tool that could be used to modify the junit.xml document and add those properties. I haven't used XSLT in many years, and never with Go, but I think it would be possible to write something using XSLT to add those properties after the file is created.

[Victor-Chirinian-Vimeo]

@dnephin Thanks for replying to my question. The data being added to the report would come from another source. We are currently integrating our Test Automation Suite results with TestRail by including a TestId in the name and parsing it out of the name attribute in the report. It would be nice to do something similar to this, where the identifier is included in the report as a testcase property:

<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="3049" name="tests.LoginTests">
    <testcase classname="tests.LoginTests" name="test_case_1" time="650">
     <properties>
         <property name="test_id" value="C123"/>
     </properties>
    </testcase>
  </testsuite>
</testsuites>

[dnephin]

Thanks for the explanation! I'd be happy to explore some options for solving this problem.

I think the few things we need to figure out are:

1. Where does the test_id value come from?
2. How does gotestsum store the association of test name (including package) to test_id? It might be nice to make this generic metadata associated with a test case instead of only test_id.
3. How would someone configure gotestsum with the xml tags and attributes to render the metadata.

I've got some initial thoughts, but I'm hoping we'll find other options too.

Where does the test_id value come from?

Two options that come to mind:

1. gotestsum could parse the output of tests and look for some kind of identifier that communicates the metadata (ex: GOTESTSUM_METDATA=test_id=<value>). This would require that every test case print the ID. Maybe it would be possible to print the full xml to be added to the report? In some ways this is similar to how github actions accepts values from commands.
2. gotestsum could read key=value pairs from a file, or maybe even key=<xml> pairs, where key is the package and test name. Maybe a JSON document would be a better option than key=value pairs. If the file can contain the full xml to inject for each test case, that would solve problem 3.

What do you think of these options? Is there a better way to communicate this metadata?

I think the other two problems will largely depend on how we solve this first one.

[Victor-Chirinian-Vimeo]

1. The Test Id's are generated by TestRail, which is a test case management system. They are used to report results from automated tests using their API. I can imagine folks would/could use this or any other system to report test results to, if they are not running unit tests.
2. I agree, storing metadata for each test should be generic and not specific to test_id.
3. Could we expose a package that allows the individual test to edit or add data to the generated JUNIT xml file ? Is the XML file generated at the end of execution ? Ideally it would be great if we could add it in a test setup or teardown method.

[dnephin]

Could we expose a package that allows the individual test to edit or add data to the generated JUNIT xml file ? Is the XML file generated at the end of execution ? Ideally it would be great if we could add it in a test setup or teardown method.

That's an interesting idea! The XML file is generated at the end of execution. The challenge is in how the tests are run. The processes involved look something like this:

flowchart LR
gotestsum -->|run| gotest["go test"]
gotest -->|build| binary["pkg.test"]
gotest -->|run| binary
binary -.->|text output| gotest
gotest -.->|json output| gotestsum

Loading

pkg.test is the compiled binary that actually runs the tests. gotestsum and go test both run in separate processes from pkg.test and the only standard way to receive output from the test is from t.Log or fmt.Print sent to stdout from the pkg.test process.

It is also technically possible that we could call the pkg.test binary with extra flags that instruct it to write output to a specific file, and have gotestsum read that file to access the metadata.

These are roughly the two options I mentioned at the end of #311 (comment)

gotestsum could definitely provide a package to make both of these things easier, but the package would effectively either be printing to stdout, or writing to a file, and gotestsum would still need to be given the name of that file.

From the test suite side a test setup or teardown method could definitely work with both of these options. Either to print the test_id to stdout, or to write it to a file.

Does a test named tests.LoginTests always have the same test_id, or does it get a different one on each run?

---

So far I'm thinking that the file approach would be easier. It should be easier to discover the feature, and less likely to cause problems for existing test suites.

Your test suite would write out a line delimited JSON file, one line per test. TestMain can be used to write the file to disk right before the test program exits. The file format would look something like this:

{"Package": "tests", "Test": "LoginTests", "JunitXMLProperties": "<properties><property name="test_id" value="C123"/></properties>"}
...

You would run gotestsum --test-metadata=meta.json, where meta.json is the file described above. gotestsum would parse the file, and add the JunitXMLProperties to the appropriate section of the junit.xml when it writes out the file.

[Victor-Chirinian-Vimeo]

@dnephin This sounds great, thanks for writing out the flow diagram. I think you are right, given what you have already mentioned, an external metadata file would be the best approach. Can we have the JunitXMLProperties be an array of json objects that are just {"name": "value"}, instead of actually including the xml tags ?

[dnephin]

That is definitely possible. I like the idea of accepting XML tags because I'm not sure what other systems might expect for the xml tags and attributes. Accepting XML would allow this feature to be used by other systems, but definitely does put more of a burden on the user to ensure they are producing valid XML.

Maybe the library that is responsible for creating this file could take on the responsibility of ensuring valid XML is written to the file? Are there other problems with this approach that I've missed?

[Victor-Chirinian-Vimeo]

No, I think this is great, and appreciate you taking the time to respond and comment on this enhancement. Please let me know if you need anything else from me. I would be happy to help test this enhancement when it becomes available.

[dnephin]

There is a Go proposal open (golang/go#43936) to Allow Go Tests to Pass Metadata. If this is accepted this would be a great solution to the problem. The test suite could write the metadata, and gotestsum could read it and add it to the Junit.xml. All that would be needed is a convention about how to structure the metadata.