System.Private.Xml tests are too particular about the installed .NET Core version and Unable to generate code coverage report with coverlet

[mrj001]

I'm using:

• Windows 10 Home build 18363.
• Visual Studio 2019 16.5.1 and 16.6.0 Preview 1 (both are Community Edition)
• git version 2.25.1.windows.1

I'm building using the git bash command prompt, as follows:

$ git clean -xdf
$ git pull upstream master
$ git push origin master
$ ./build.cmd -subsetCategory coreclr-runtime -c Checked
$ ./build.cmd -subsetCategory coreclr-libraries -runtimeConfiguration Checked
$ ./build.cmd -buildtests -runtimeConfiguration Checked
- finishes with "29 Errors"
$ cd src/libraries/System.Private.Xml
$ ../../../dotnet.cmd test -p:CollectCoverage=true -p:CoverletOutput="/c/Users/mrj00/source/repos/runtime/" -p:MergeWith="/c/Users/mrj00/source/repos/runtime/coverage.json" -maxcpucount:1 -p:CoverletOutputFormat=\"json,opencover\"

This resulted in numerous errors like this:

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
Testhost process exited with error: It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '5.0.0' was not found.

• The following frameworks were found:
  5.0.0-preview.3.20169.1 at [C:\Users\mrj00\source\repos\runtime.dotnet\shared\Microsoft.NETCore.App]
  You can resolve the problem by installing the specified framework and/or SDK.
  The specified framework can be found at:
• https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=5.0.0&arch=x64&rid=win10-x64
  . Please check the diagnostic logs for more information.

Test Run Aborted.

To address the excessive pickiness of the framework version, I did this:

$ cd ../../../.dotnet/shared/Microsoft.NETCore.App
$ ln -s 5.0.0-preview.3.20169.1 5.0.0
$ cd ../../../src/libraries/System.Private.Xml
$ ../../../dotnet.cmd test -p:CollectCoverage=true -p:CoverletOutput="/c/Users/mrj00/source/repos/runtime/" -p:MergeWith="/c/Users/mrj00/source/repos/runtime/coverage.json" -maxcpucount:1 -p:CoverletOutputFormat=\"json,opencover\"

The tests now run, but there is no output indicating that a coverage report is being generated.

$ find ../../.. -name coverage.json

No file was found.
Neither does there appear to be a file for the opencover format in the runtime folder.

Sorry, but I've really got two issues in one here:

1. the failure to find a version of the framework that should have been found (per global.json).
2. the failure to generate a coverage report.

[mrj001]

@ViktorHofer Pursuant to @danmosemsft 's reply to my above mention of this issue referring me to the instructions at:https://github.com/dotnet/runtime/blob/master/docs/workflow/building/libraries/code-coverage.md

These instructions cover the case of generating a report on the entire repo, or a single project.

I was able to follow the instructions for this single project:

src/libraries/System.Private.Xml/tests/XmlSchema/XmlSchemaSet/System.Xml.XmlSchemaSet.Tests.csproj

This worked. However, the output was generated in

artifacts\bin\System.Xml.XmlSchemaSet.Tests\netcoreapp5.0-Debug\report\index.htm

rather than the "aforementioned", which was:

artifacts\coverage\index.htm

No great problem with that - the output of the process clearly indicated the correct location.

However, my main problem is that I'm trying to generate a coverage report based upon all the test projects in one solution:

src/libraries/System.Private.Xml/System.Private.Xml.sln

and this is where I ran into difficulty. Is there a way to do this?

[danmoseley]

rather than the "aforementioned",

Mind offering a PR to fix the doc :)

[mrj001]

@danmosemsft
Created PR #34340.

[ericstj]

With respect to the pickiness -- we are building the shared framework here and the tests need to run on the version we built. It's in the output under artifacts/bin/testhost and has version 5.0.0.

@ViktorHofer can you have a look and see what we need to do to fix coverage?

[ViktorHofer]

@mrj001 a few observations:

1. We don't support dotnet test, instead use dotnet build /t:Test.
2. We don't support collecting code coverage on an sln. This is mainly because of 1. Either invoke dotnet build /t:Test /p:Coverage=true on a single csproj or run all tests from root with build.cmd/sh libs.tests -test -coverage. If you just want to collect System.Private.Xml you need to change the glob pattern in tests.proj: https://github.com/dotnet/runtime/blob/master/src/libraries/tests.proj#L16 to $(MSBuildThisFileDirectory)System.Private.Xml\tests\**\*.Tests.csproj.

With #945 dotnet test will just work and code coverage measurement will be possible on slns.

[mrj001]

@ericstj, @ViktorHofer
Closing, as I believe this has addressed all the points I raised.