VSTest with coverlet collector fails

[arabelaa]

I have a .NETCore 3.1 test project with "Microsoft.NET.Test.Sdk" 16.7.1 of and "coverlet.collector" 1.3.0.

I am getting an exception when running tests using VSTest (I can't switch to dotnet test approach, unfortunatly - there is a strong requirement to stick to vstest)

vstest.console.exe --collect:"XPlat Code Coverage" C:\Projects\test\Web.Tests\bin\Debug\Web.Tests.dll /TestAdapterPath:C:\.nuget\packages\coverlet.collector\1.3.0\build --Diag:"C:\Projects\diag.txt"

I get the following warning in the console:

Data collection : Unable to find a datacollector with friendly name 'XPlat code coverage'.
Data collection : Could not find data collector 'XPlat code coverage'

And in diag.txt file I see:

datacollector.exe, TestPluginDiscoverer: Failed to get types from assembly 'coverlet.collector, Version=1.3.0.0, Culture=neutral, PublicKeyToken=4beb1ad7f05a1c71'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
TpTrace Warning: 0 : 60664, 1, 2020/11/14, 22:42:00.029, 8944115731653, datacollector.exe, LoaderExceptions: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

I followed the discussion mentioned on microsoft/vstest#2278 but it seems like a separate issue.
It feels I am missing some type of configuration.

I can't figure out the source of the problem. Any help is highly appreciated. Thanks in advance.

[MarcoRossignoli]

Do you have net4x target framework on your projects?

Can you try with the temporary version on my "debug" feed? https://f.feedz.io/marcorossignoli/coverletunofficial/nuget/index.json 3.0.0-preview.7 version

[arabelaa]

I added the Framework parameter --Framework:".NETCoreApp,Version=v3.1"

and now I am getting

TpTrace Error: 0 : 63980, 4, 2020/11/16, 12:28:41.887, 10304255399683, testhost.dll, InProcDataCollectionExtensionManager: Error occurred while loading the InProcDataCollector : coverlet.collector.dll , Exception Details : System.ArgumentException: Absolute path information is required. (Parameter 'assemblyPath')
   at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformAssemblyLoadContext.LoadAssemblyFromPath(String assemblyPath)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollector.LoadInProcDataCollectorExtension(String codeBase)
TpTrace Information: 0 : 63980, 4, 2020/11/16, 12:28:41.887, 10304255403083, testhost.dll, AssemblyResolver.AddSearchDirectories: Adding more searchDirectories 
TpTrace Error: 0 : 63980, 4, 2020/11/16, 12:28:41.888, 10304255409679, testhost.dll, InProcDataCollectionExtensionManager: Error occurred while Initializing the datacollectors : System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollector.CreateObjectFromType(Type type)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollector.LoadDataCollector(IDataCollectionSink inProcDataCollectionSink)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager.CreateDataCollector(String assemblyQualifiedName, String codebase, XmlElement configuration, TypeInfo interfaceTypeInfo)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager.InitializeInProcDataCollectors(String runSettings)

[MarcoRossignoli]

The error above seem related to .NET Framework test process that tries to load .NET Core coverlet collectors. This is not supported in coverlet until version 3.0.0(not released yet).
Since next version collectors will be a netstandard one so should be usable by a .NET Framework runtime(.NET core assemblies are different from .NET Framework one, so the resolution errors).

[arabelaa]

I used the nuget from the suggested feed https://f.feedz.io/marcorossignoli/coverletunofficial/nuget/index.json and I no longer see the exception.

But now I have other errors with datacollector.exe (our tests create a kestrel process and we assert the response we get from an API endpoint)

TpTrace Warning: 0 : 61228, 1, 2020/11/16, 19:08:19.453, 10544031072556, datacollector.exe, [coverlet]Unable to resolve method reference "System.Threading.Tasks.Task`1<!0> System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Collections.Generic.IList`1<AssetDto>>::get_Task()", assuming calls to will return
TpTrace Verbose: 0 : 61228, 1, 2020/11/16, 19:08:19.453, 10544031075300, datacollector.exe, [coverlet]TryWithCustomResolverOnDotNetCore for System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
TpTrace Verbose: 0 : 61228, 1, 2020/11/16, 19:08:19.453, 10544031075781, datacollector.exe, [coverlet]Not a dotnet core app

------------------------------------------------------------------------------------------------------------------------------------
TpTrace Warning: 0 : 61228, 1, 2020/11/16, 19:08:24.188, 10544078419027, datacollector.exe, [coverlet]Unable to resolve method reference "!0 System.Nullable`1<System.DateTime>::GetValueOrDefault()", assuming calls to will return

[arabelaa]

Looking over #991 I realized that you don't support API tests (integration tests) coverage. And this will be released in 3.0.0 release. But it still feel strange to see this errors.

[MarcoRossignoli]

Seems that there is an issue resolving one assembly System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
You app is not a .NET Core one so coverlet cannot try to resolve browsing SDK installed on machine, can you try to copy that assembly on publish folder?

Last error on GetValueOrDefault() is a warning no prob for it.

[arabelaa]

Our app is a .NET Core application. The problem is that the system under test is an API which is hosted in kestrel. And this is not yet supported by coverlet. Do you have a release date for 3.0.0 version?

[MarcoRossignoli]

The problem is that the system under test is an API which is hosted in kestrel. And this is not yet supported by coverlet.

It depends, if the test process opens a port and there some call on it, should work. I mean a scenario like dotnet test where the test open port and some client inside test call on that should work with no problem. Can you describe your scenario?

[arabelaa]

Our test process launches a dotnet process using ProcessStartInfo.Start with some launchSettings arguments

 "profiles": {
    "Kestrel": {
      "commandName": "Project",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      },
      "use64Bit": true,
      "applicationUrl": "https://localhost:6000"
    } 

[MarcoRossignoli]

Ok if the process hosts instrumented assemblies should work..I mean if test keeps the process alive and shutdown at the end and the ProcessExit will fire hits should be collected. Btw it's hard to understand without a complete picture, if you can create a "repro" I can take a look.

[arabelaa]

The test project has integration tests (that call a web application - which is hosted in IIS or Kestrel depending on a configuration).

First question: Is IIS scenario supported? There is a configuration in .runsettings that allows to collect from childProcesses.

<CollectFromChildProcesses>True</CollectFromChildProcesses>
 <!-- When set to True, restarts the IIS process and collects coverage information from it. -->
<CollectAspDotNet>True</CollectAspDotNet>

For Kestrel we start the process when the test starts. Something along this lines

var url = "http://localhost:2345";
string dotnetDir = Environment.GetEnvironmentVariable("DOTNET_INSTALL_DIR") ?? Path.Combine(Environment.GetEnvironmentVariable("ProgramW6432"), "dotnet");
 var dotnet = Path.Combine(dotnetDir, "dotnet.exe");
 var project = Path.Combine("src", "Web", "Web.Host", "bin", "Web.dll");
 var processToStart = new Process
 {
   StartInfo = new ProcessStartInfo()
    {
       FileName = dotnet,
       Arguments = $@"{project} --urls ""{url}""",
       UseShellExecute = false,
       CreateNoWindow = false
    }
 };
 processToStart.Start();

When running
.\vstest.console.exe /settings:C:\CodeCoverage.runsettings C:\bin\Debug\Web.Tests.dll /diag:C:\Projects\Diag.txt /TestAdapterPath:C:\.nuget\packages\coverlet.collector\3.0.0-preview.9\build

we see no code coverage for the API's.
Is my understanding correct that with 991 the behavior I mentioned will be supported? Or this should already work and I am missing some configuration?

[MarcoRossignoli]

Can you share /diag files(be sure to not share secrets)
Can you also enable tracker logger to understand how hits are collected https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#enable-injected-tracker-log

Is my understanding correct that with 991 the behavior I mentioned will be supported? Or this should already work and I am missing some configuration?

Will be supported but not through dotnet test command you should publish file somewhere and after use .NET tool to spin up process that host asm to cover.
Take a look here https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/GlobalTool.md#code-coverage-for-integration-tests-and-end-to-end-tests

It's not related to host process it's related to instrumentation phase, I try to explain better how coverlet workflow works:

1. in case of collectors/msbuild integration msbuild or out of process collectors(orchestrated by vstest plat) will instrument all dll(applying passed parameters).
2. host process load dll and run tests
3. when ProcessExit will fire for test host process hits will be persisted to disk.

So it's not a problem the "host process" if ProcessExit event will fire on it...if for some reason a "kill" is used no hits will be collected.

Your use case is particular, I mean you do integration using a test framework(usually I mock server for instance using https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-5.0#aspnet-core-integration-tests). Anyway coverlet can collect hits from concurrent process and also with old app domain, so the only problem I see is a premature kill of process(if you kill after that processToStart.Start(); at the end of test and not shutdown it gracefully, kill process prevent process exit event to fire).

[arabelaa]

Let me see if I got this right.

If I have a .NET Core app hosted in IIS(w3wp.exe)/kestrel and I have a suite of integration tests I should be able to collect code coverage using coverlet (the preview version which support .netstandard targetframework) - because the IIS process has the dll&pdb and they are available for instrumentation - even if the IIS process was no started by the test process.

I used the tracker logger but the file was not generated for the module I was interested in instrumenting (it's related to the amount of errors in diagnostics file).
I've attached the logs from host and datacollector after running the vstest command (with IIS process - that was started upfront and not during the test execution).
Note: the same behavior I have with kestrel.

diag.host.20-11-27_11-09-50_76174_4.txt
diag.datacollector.20-11-27_11-09-24_20487_4.txt

[MarcoRossignoli]

Ok thanks for help on investigation...I see only 2 asm instrumented

TpTrace Verbose: 0 : 11588, 1, 2020/11/27, 11:09:28.725, 3383297749099, datacollector.exe, [coverlet]Instrumented module: 'C:\Projects\Testing\test\Web.Tests\bin\Debug\Resources.dll'
TpTrace Verbose: 0 : 11588, 1, 2020/11/27, 11:09:25.403, 3383264537040, datacollector.exe, [coverlet]Instrumented module: 'C:\Projects\Testing\test\Web.Tests\bin\Debug\System.Reactive.Linq.dll'

And a lot of resolution errors

TpTrace Verbose: 0 : 11588, 1, 2020/11/27, 11:09:28.746, 3383297961850, datacollector.exe, [coverlet]TryWithCustomResolverOnDotNetCore for System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Also I don't see in-process collector loaded(host log file).

Are you able to provide a similar repo?I think it's a corner case and need to investigate further.
Or you can try with beta package and coverlet tool https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-5.0#aspnet-core-integration-tests but it won't follow the dotnet test workflow so you need to change how you're doing the integration tests.

[arabelaa]

Thanks for looking into it.

I've attached a simplified example Scenario.zip that reproduces the issue. In IntegrationTests project is visible how I am testing the API using dotnet process (using vstest).

.\vstest.console.exe /settings:C:\source\repos\Scenario\IntegrationTests\CodeCoverage.runsettings C:\source\repos\Scenario\IntegrationTests\bin\Debug\netcoreapp3.1\IntegrationTests.dll /diag:C:\source\repos\Scenario\IntegrationTests\diagnostics.txt /TestAdapterPath:C:\.nuget\packages\coverlet.collector\3.0.0-preview.9\build

While creating the repo I realized that the test process is not aware of the dll that needs instrumentation (because it's an API and there is no direct reference in the test project).
And I copied the dll that I wanted to be instrumented inside the test bin folder and the issues with TryWithCustomResolverOnDotNetCore popped up.

I am inclined to say that my scenario is not supported. And I don't think I can use coverlet tool - because my tests are not an executable (they are plain XUnit tests).

[MarcoRossignoli]

Thanks for repro, I'll take a look asap.

[MS-Manikandan]

Hi @MarcoRossignoli, I too get the same error when I try to get coverage for my .netcore3.1 test project I have also added coverlet.collector and coverlet.msbuild version 3.1.0 as package reference on the unit test project,the .netcore test project has 2 .netstandard2.0 projects as reference, but didn't resolve it.

&"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" --collect:"XPlat Code Coverage" netcoreapp3.1\UnitTest.dll /TestCaseFilter:"Name=TestName"

Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collection : Unable to find a datacollector with friendly name 'XPlat Code Coverage'.
Data collection : Could not find data collector 'XPlat Code Coverage'
NUnit Adapter 4.2.0.0: Test execution started

[MarcoRossignoli]

@Manikandanmani I think that in your case the issue is that you didn't set the /TestAdapterPath if used with vstest.console.exe test platform needs to know where to discover the extensions and coverlet is not shipped together with the plat self.
You should point to the folder that contains the coverlet collector like /TestAdapterPath:C:\Users\Marco\.nuget\packages\coverlet.collector\3.1.0\build\netstandard1.0

[MS-Manikandan]

Hi @MarcoRossignoli, thanks for reaching out, after adding the test adapter path the xml report got generated, but the coverage results was empty, but for another .netcore3.1 test project reference a .netstandard2.0 project without specifying the test adapter path worked, but that too gave an empty coverage file. could you let me know how to generate coverage results correctly and why does without specifying the testadapterpath worked this project?

[MarcoRossignoli]

, but for another .netcore3.1 test project reference a .netstandard2.0 project without specifying the test adapter path worked, but that too gave an empty coverage file

Depends on how the compilation is done...sometimes collector is copied inside bin folder(publish for instance) in some other scenario not, so to be sure to always load it you should specify /TestAdapterPath: when you use vstest.console.exe directly.
Test platform by design search extensions in some predefined places(I don't remember all the place) and one of these is the folder where the test container is placed, so I suspect that the coverlet extension was found here in you working sample.

if you run with dotnet test project coverlet is injected using msbuild target and so we're adding correct path to the TestAdapterPath in a transparent way.

could you let me know how to generate coverage results correctly

can you provide a repro?It's hard without it. Usually we need a repro and the command line.

[MS-Manikandan]

Depends on how the compilation is done...sometimes collector is copied inside bin folder(publish for instance) in some other scenario not, so to be sure to always load it you should specify /TestAdapterPath: when you use vstest.console.exe directly. Test platform by design search extensions in some predefined places(I don't remember all the place) and one of these is the folder where the test container is placed, so I suspect that the coverlet extension was found here in you working sample.

@MarcoRossignoli , You were right, I forgot to inform that I executed dotnet publish command, after executing dotnet publish command and ran a test using vstest.console.exe without specifying the testAdapterPath executes the tests and gives the coverage.cobertura.xml, but even performing a dotnet publish or specfying the path and running the test using vstest the coverage.cobertura.xml shows zero code coverage, do have any idea what might be missed so it gave zero coverage as result.

[MarcoRossignoli]

There're possible more causes, but I need a repro to understand what's happening, if you want to try yourself this is the way to enable and investigate what coverlet is doing https://github.com/MarcoRossignoli/coverlet/blob/master/Documentation/Troubleshooting.md#collectors-integration

[MS-Manikandan]

Hi @MarcoRossignoli, adding diag flag In vstest.console.exe /Diag:"Diag.txt" /Collect:"XPlat code coverage" UnitTest.dll /TestCaseFilter:"Name=Name" /TestAdapterPath:".nuget\packages\coverlet.collector\3.1.0\build\netstandard1.0" ran the test and generated empty results in coverage.xml, when viewing the *datacollector.txt it gave these

TpTrace Information: 0 : 24676, 1, 2022/02/17, 15:53:01.838, 213226831877, datacollector.exe, DataCollectionRequestHandler.ProcessRequests : Datacollector received message: (DataCollection.AfterTestRunEnd) -> false
TpTrace Verbose: 0 : 24676, 1, 2022/02/17, 15:53:01.855, 213226974195, datacollector.exe, [coverlet]CoverletCoverageDataCollector: SessionEnd received
TpTrace Information: 0 : 24676, 1, 2022/02/17, 15:53:01.921, 213227628215, datacollector.exe, [coverlet]CoverletCoverageDataCollector: Saved coverage report to path: 'C:\Users\XYZ\Temp\a900b1e3-1e0a-4237-95c3-bcfdf5cec5b6\coverage.cobertura.xml'
TpTrace Verbose: 0 : 24676, 1, 2022/02/17, 15:53:01.922, 213227637384, datacollector.exe, [coverlet]CoverletCoverageDataCollector: Sending attachment to test platform
TpTrace Information: 0 : 24676, 6, 2022/02/17, 15:53:01.931, 213227726291, datacollector.exe, DataCollectionAttachmentManager.AddNewFileTransfer : Copying file C:\Users\XYZ\Temp\a900b1e3-1e0a-4237-95c3-bcfdf5cec5b6\coverage.cobertura.xml to TestprojectPath\UnitTest\TestResults\29ebd1a5-220f-4d4b-a0d5-aabce8805c11\coverage.cobertura.xml
TpTrace Information: 0 : 24676, 6, 2022/02/17, 15:53:01.946, 213227868622, datacollector.exe, DataCollectionAttachmentManager.AddNewFileTransfer : Copied file C:\Users\XYZ\Temp\a900b1e3-1e0a-4237-95c3-bcfdf5cec5b6\coverage.cobertura.xml to TestprojectPath\UnitTest\TestResults\29ebd1a5-220f-4d4b-a0d5-aabce8805c11\coverage.cobertura.xml
TpTrace Verbose: 0 : 24676, 6, 2022/02/17, 15:53:01.948, 213227896239, datacollector.exe, [coverlet]CoverletCoverageDataCollector: SendFileCompleted received
TpTrace Verbose: 0 : 24676, 1, 2022/02/17, 15:53:01.955, 213227968130, datacollector.exe, [coverlet]CoverletCoverageDataCollector: Deleted report directory: 'C:\Users\XYZ\Temp\a900b1e3-1e0a-4237-95c3-bcfdf5cec5b6'
TpTrace Verbose: 0 : 24676, 1, 2022/02/17, 15:53:01.956, 213227992991, datacollector.exe, Test Attachment Description: Collector:'XPlat code coverage' Uri:'datacollector://microsoft/CoverletCodeCoverage/1.0' Description:'' Uri:'file:///TestprojectPath/UnitTest/TestResults/29ebd1a5-220f-4d4b-a0d5-aabce8805c11/coverage.cobertura.xml'
TpTrace Information: 0 : 24676, 1, 2022/02/17, 15:53:01.956, 213228004958, datacollector.exe, DataCollectionManager.CleanupPlugins: CleanupPlugins called
TpTrace Verbose: 0 : 24676, 1, 2022/02/17, 15:53:01.956, 213228005864, datacollector.exe, DataCollectionManager.CleanupPlugins: Cleaning up 1 plugins
TpTrace Verbose: 0 : 24676, 1, 2022/02/17, 15:53:01.956, 213228013239, datacollector.exe, dataCollectorInfo.DisposeDataCollector: calling Dispose() on Coverlet.Collector.DataCollection.CoverletCoverageCollector
TpTrace Verbose: 0 : 24676, 1, 2022/02/17, 15:53:01.956, 213228019720, datacollector.exe, [coverlet]CoverletCoverageDataCollector: Disposing

here both the files produced has empty results, when I added the runsettings file along with above command the test ran, but it didn't generate any coverage file. I refernced the coverlet.runsettings file from https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md, project packages , even tried dotnet pack and publish didn't work either.

DataCollectionAttachmentManager.AddNewFileTransfer : Copying file C:\Users\XYZ\Temp\a900b1e3-1e0a-4237-95c3-bcfdf5cec5b6\coverage.cobertura.xml to TestprojectPath\UnitTest\TestResults\29ebd1a5-220f-4d4b-a0d5-aabce8805c11\coverage.cobertura.xml

[MarcoRossignoli]

@Manikandanmani can you take a look at every log starting with [coverlet] inside collector file log?You should see how many and what dll are instrumented for the coverage.

[github-actions]

This issue is stale because it has been open for 3 months with no activity.

[github-actions]

This issue was closed because it has been inactive for 275 days since being marked as stale.