[junit] Add a file attribute to xunit2

[webknjaz]

Currently linked schema https://github.com/jenkinsci/xunit-plugin/blob/xunit-2.3.2/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd#L127 allows a file attribute in <testsuite> nodes. It's currently missing but would be very useful.

AFAICS junitxml plugin dumps everything under a single testsuite node. It was probably fine with xunit1 that allowed per-testcase file attribute to be set. Now that xunit2 is going to be a default, it needs a similar feature too.

But since this attribute is per-suite in xunit2, the plugin will have to separate <testcase>s into multiple <testsuite>s.

Why is this important to me personally? I usually set up test reporting in CIs that support it (AppVeyor and Circle CI, for example). And it's nice to see where the test is coming from.

I wanted to turn the test report into GitHub Actions annotations from XML using a simple action that requires almost zero effort to set up (shyim/junit-report-annotations-action#3). But I hit a roadblock when I realized that for putting annotations on lines in files, it has to know what those files are.

[bluetech]

Disclaimer: I don't actually use junitxml.

This sounds good to me, it does look like some indication of where the test comes from is missing.

The testsuite attributes we might want to use for it are file, package, id, group.

Do you happen to have an example of file from another project/language (e.g. Java) which uses it? I'd be interested to see the granularity in which they define their testsuites. Does each file become a testsuite? Or package? Are they listed in a hierarchy or flat? etc.

One thing to note is that pytest has the record_testsuite_property fixture which is documented to apply to the "root testsuite". But I see that testsuite elements can be nested, so it's not a problem, just means we must retain a single one at the root.

[webknjaz]

@bluetech unfortunately I don't have better examples from other languages. I've only tried it with pytest and those CIs that I listed earlier. btw xunit2 reports seem to be less usable there but I haven't checked closer.

Thanks for pointing to record_testsuite_property, I haven't thought about it.
FWIW I solved my immediate problem by downgrading to xunit1 (that the referenced GHA supports, apparently). For anybody curious, here's how the outcome looks: https://github.com/ansible/ansible-lint/pull/910/files/2c21181c47aac927f8c77135b2ba3fade051d0f9#diff-586e1a30597f4eb8296785ca7b4b638eR214.
It turned out to be a poor integration so I'm considering creating a native pytest plugin that would emit annotations in GHA workflows but it's still important that this junitxml issue is documented.

[webknjaz]

FTR here's an example of how AppVeyor reporting of xunit1 vs xunit2 differ:

1. xunit1 https://ci.appveyor.com/project/njsmith/trustme/builds/34322432/job/rygsb2do5tkmtmxc/tests
2. xunit2 https://ci.appveyor.com/project/CherryPy/cheroot/builds/34315268/job/yosp77mbywt35urh/tests

They use the junit_suite_name value instead of the filename when processing xunit2 while with xunit1 they display the file name in that column...

[webknjaz]

So the new Codecov interface for displaying the test result statistics (#12689) has integration instructions that recommend using legacy (junit1) test reports, supposedly for the same reason: https://app.codecov.io/gh/pytest-dev/pytest/tests/new.