Indicate framework in produced xml file

[thj-dk]

I have a xUnit test project with multiple TargetFrameworks (netcoreapp3.1 and net471), and I'm generating multiple test results files using LogFilePath={assembly}-{framework}.xml.

However, when importing these files using Jenkins JUnit plugin, the framework information is lost and we're unable to differentiate between test execution frameworks.

Do you know of any workaround? Maybe an option that makes it possible to specify/modify the testsuite name in the produced xml?

[Siphonophora]

Hi @thj-dk

Sounds like their plugin is looking at the TestSuite Name element when grouping or labeling results?

I think a config flag would probably be the right way to go. Let me think about the cleanest option, but that wouldn't be difficult.

You may have noticed we have a page explaining how to customize the xml output using existing config options to make these reports more readable in the GitLab UI. https://github.com/spekt/junit.testlogger/blob/master/docs/gitlab-recommendation.md If you want to help us put together something similar for Jenkins, that would be great.

[thj-dk]

I have made some verification, and it turns out that the grouping is a combination of TestSuite name and package, and TestCase name from what I can see. So some way of prefixing both would be desired.

I also found a different bug. The time attribute of the testcase element has a incorrect decimal separator that isn't understood by Jenkins time="0,0035817" where the testsuite element has the correct separator time="0.0035817"

If we can get this fixed I would be happy to contribute with a Jenkins-guide.

[Siphonophora]

To clarify, when you say both, do you mean TestSuite name and package? Maybe a short example file would be good.

For the comma vs decimal, that's clearly a Culture issue. Do you know if the culture these are running under is "da-DK"? I can try to repo and get this fixed.

[thj-dk]

In order to make everything show up correctly in Jenkins (we're using an older version here - 2.46), I have to prefix TestSuite name and package attributes with framework name (e.g. NETCoreApp31 and NETFramework471).

The server culture is either da-DK or nb-NO, so in that regard, the comma decimal separator is correct. However I do not think that Jenkins understands that - only period. I do believe that these logger files should always be generated using period (just like the testsuite@time attribute is).

[Siphonophora]

That makes sense. I will let you know when there is a preview version for you to try

[thj-dk]

I made some further experiments, and it turns out that specifying the testsuite name is actually enough for Jenkins to interpret the test files correctly.
I've made a powershell script that modifies the generated files and sets the testsuite@name to the executing framework.

Then the tests shows up in Jenkins as e.g. AssemblyName.dll.NETFramework471

So I guess that an options for specifying the testsuite@name is all that is required after all. And with support for placeholders, so we can do something like TestSuitePackageName={framework}.

[Siphonophora]

@thj-dk I added a command line flag TestSuiteFormat=AddFramework to address the mult-targeting issue. The pre-release version is here: https://www.myget.org/feed/spekt/package/nuget/JunitXml.TestLogger/3.0.93

[thj-dk]

@Siphonophora That is great, thank you so much. We can definitely make great use of this.

However, did you consider just making an option for specifying the test-suite name directly? It doesn't have to contain the full assembly name. I think there could be many uses for setting this property to something entirely different, when one needs to be running tests in multiple contexts/environments etc.

The default value could be the assembly name like it is today, but then make it configurable and let it support placeholders. So we can do TestSuiteName={assemblyname}.{framework] or e.g. TestSuiteName={framework}-staging

I can give it a shot if you want me to.

[Siphonophora]

@thj-dk The other option I thought about was to give an option where the file name would be used for the Test Suite name.

I thought dotnet test --logger:"junit;LogFilePath={assemblyname}.{framework}.xml;TestSuiteName={assemblyname}.{framework}.xml" would be a bit cumbersome or even confusing if they ended up out of sync.

I wouldn't mind an option like this: dotnet test --logger:"junit;LogFilePath={assemblyname}.{framework}.xml;TestSuiteFormat=UseFilename" . How does this sound? My initial thought is that these names would usually end up being the same. Can you think of a situation where this wouldn't work?

[thj-dk]

I cant really figure out what the intended format here is... I'm not entirely sure that the file name has to be part of the test suite name? I haven't found any documentation describing that as a requirement. The testsuite@package attribute contains the name of the assembly file. The tests are easily identifiable through this and the testcase element.

In Jenkins UI, the tests are grouped by: testsuite@package > testcase@classname > testcase@name
So right now that is AssemblyName > AssemblyName.ClassName > AssemblyName.ClassName.MethodName.TestName
When viewing the tests results, the test "headline" description is (from the XML):

testcase@classname.testcase@name (from testsuite@package.testsuite@name)

Translated, that is currently:

AssemblyName.ClassName.AssemblyName.ClassName.MethodName.TestName (from AssemblyName.dll.AssemblyName)

Which is horrible to read.

To fix this, I've made a Powershell script, that ends up doing this to the produced XML:

# Set testsuite@name to $framework.
$xml.SelectNodes("//testsuite") | %{
  $_.SetAttribute("name", $framework);
}

# Remove full class path from testcase@name and append [$framework] to easily identify the executing framework.
$xml.SelectNodes("//testcase") | %{
  $_.SetAttribute("name", $_.GetAttribute("name").Replace($_.GetAttribute("classname") + ".", "") + "[$framework]");
}

This procuces the following views in Jenkins:

I'm not sure what the correct answer is...

[Siphonophora]

@thj-dk I didn't mean the file name and test suite name needed to be the same. I was thinking we might be simpler if we had an option for them to be the same.

For the rest of your example, it looks like we need some more formatting options to make the report readable in Jenkins. Let me make up a little test suite we can use as an example, so we can look at the same xml output.

For my notes. You are using https://plugins.jenkins.io/junit/ right?