verify how badly we got junit-xml wrong

[RonnyPfannschmidt]

according to the relax Schema's at http://help.catchsoftware.com/display/ET/JUnit+Format
we have quite a lot of the elements we should have at the suite level at a testcase level

[dnozay]

Hi, I am currently maintaining unittest-xml-reporting https://github.com/xmlrunner/unittest-xml-reporting.

I don't have a solution to the issue, but I was trying to figure out the open issues with the junitxml plugin and see if (license issues aside) there is some collaboration that can happen. Maybe the junitxml plugin can leverage unittest-xml-reporting; contributions from some of you would be welcome too.

[RonnyPfannschmidt]

I am interested in completely externalizing the junitxml plugin in the pytest 3.x series

[nicoddemus]

Other resources:

https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

http://www.utilities-online.info/xsdvalidation

[RonnyPfannschmidt]

according to the xsd, im reasonably sure we got the properties wrong

[jrd]

According to this https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd
skipped property is mandatory in testsuite tag.
This definition schema is the one used in jenkins junit/xunit plugins

[gtmanfred]

Any chance of getting this fixed? We are currently having to downgrade xunit in jenkins because of this.

Thanks,
Daniel

[RonnyPfannschmidt]

@gtmanfred nobody is working on it

[icemac]

There is also #1862 which seems to describe the same problem.

[ringods]

The main thing I find missing in the currently output XML is that the top-level testsuites (plural!) tag is missing. The Schema allows for that and it would already render more sections in tools that post process the XML file, e.g. ReportPortal.

Current:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite disabled="" errors="" failures="" hostname="" id=""
           name="" package="" skipped="" tests="" time="" timestamp="">
    <testcase assertions="" classname="" name="" status="" time=""></testcase>
    <testcase assertions="" classname="" name="" status="" time=""></testcase>
    ...
</testsuite>

Allowed:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites disabled="" errors="" failures="" name="" tests="" time="">
    <testsuite disabled="" errors="" failures="" hostname="" id=""
               name="" package="" skipped="" tests="" time="" timestamp="">
        <properties>
            <property name="" value=""/>
        </properties>
        <testcase assertions="" classname="" name="" status="" time="">
            <skipped/>
            <error message="" type=""/>
            <failure message="" type=""/>
            <system-out/>
            <system-err/>
        </testcase>
        <system-out/>
        <system-err/>
    </testsuite>
    <testsuite ...>
       ...
    </testsuite>
    ...
</testsuites>

Source: https://stackoverflow.com/a/9691131/7269988

[nicoddemus]

I think this can now be closed given the introduction of junitxml_family option.

[danilomendesdias]

@nicoddemus record-property still add properties inside testcase which seems to be wrong (it should be inside testsuite).
Hypothesis is currently making usage of it and it is breaking junit reports on jenkins (see HypothesisWorks/hypothesis#1935).

[nicoddemus]

Indeed that seems to be wrong:

https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd

@danilomendesdias would you like to contribute a PR with that fix?

[danilomendesdias]

I'll work on that @nicoddemus. Thanks for you reply.