dotnet test --collect:"code coverage" includes 3rd party dlls

[GoFightNguyen]

Description

.NET Core 2.1
MSTest.TestAdapter v1.4.0
MSTest.TestFramework v.1.4.0
Microsoft.NET.Test.Sdk v 15.9.0

Steps to reproduce

Create a .NET Core Test project with a test
Add a dependency to the FluentAssertions NuGet pkg
dotnet test --blame --collect:"Code Coverage"
Open the coverage file

Expected behavior

I would not see coverage details for FluentAssertions

Actual behavior

I see coverage details for FluentAssertions

[GoFightNguyen]

I discovered the .runsettings file. At the solution level, I created a codecoverage.runsettings file. I then run this command:
dotnet test --blame --collect:"Code Coverage" --settings:..\codecoverage.runsettings

This appears to give me what I need. Here are my runsettings. Hopefully I'm not leaving out anything important.

<?xml version="1.0" encoding="utf-8"?> <RunSettings> <DataCollectionRunSettings> <DataCollectors> <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Culture=neutral"> <Configuration> <CodeCoverage> <!-- Match assembly file paths: --> <ModulePaths> <Include> <ModulePath>.*\mydomains.*.dll</ModulePath> </Include> <Exclude> <ModulePath>.*tests.dll</ModulePath> </Exclude> </ModulePaths> </CodeCoverage> </Configuration> </DataCollector> </DataCollectors> </DataCollectionRunSettings> </RunSettings>

[nigurr]

@GoFightNguyen Currently we do instrument all dependent dlls (whether they come from 3rd part dependency or nuget reference).
In this case we try to collect code coverage for FluentAssertions.

However this package can be excluded from coverage report by modifying the runsettings as you have mentioned.
This is working as designed :)

[gioce90]

Exclusion by .runsettings file is a bit painful. Exclude 3rd party and Tests assemblies it would be nice.

[BradleyBarnett]

Agree 99% of the time no one would want to include nuget packages or test assesmblies, this should be default behaviour.

[cliffAtOsaic]

I agree 100% the default behavior should not include extra DLLs. Futhermore, filtering by .runsettings file is not working for both "dotnet test" and "dotnet-coverage". This should be reopened and the default behavior should be changed. Alternatively, an easy switch should be added to remove 3rd party imported NuGets/DLLs etc.

[LoupaLoupa]

This should be reopened as a feature request, I can't fathom reasons the default behavior should be to include assemblies we have 0 control over

[cliffAtOsaic]

@GoFightNguyen Currently we do instrument all dependent dlls (whether they come from 3rd part dependency or nuget reference). In this case we try to collect code coverage for FluentAssertions.

However this package can be excluded from coverage report by modifying the runsettings as you have mentioned. This is working as designed :)

This is not working as designed, because the runsettings does not exclude extra DLLs even when explicitly told to do so.

(Nor does this make any sense to have this as the default behavior.)

[hughbidone]

Please reopen this as other already mentioned, it makes no sense to include the coverage for the source files we have no control over i.e the 3rd party packages

[nohwnd]

@fhnaseer is there anything that can be done to exclude 3-rd party dlls automatically or to help in this area?

[fhnaseer]

You can try to restrict symbols search paths to skip third party dependencies. You can try the following options in the runsettings file.

Name	Values	Default	Description	Example
SymbolsRestrictOriginalPathAccess	True, False	False	Determines if looking for a .pdb file in the original debug directory.	<SymbolsRestrictOriginalPathAccess>True</SymbolsRestrictOriginalPathAccess>
SymbolsRestrictReferencePathAccess	True, False	False	Determines if looking for a .pdb file is allowed in the path where the .exe file is located.	<SymbolsRestrictReferencePathAccess>True</SymbolsRestrictReferencePathAccess>
SymbolsRestrictDBGAccess	True, False	False	Determines if looking for debug information is allowed from .dbg files.	<SymbolsRestrictDBGAccess>True</SymbolsRestrictDBGAccess>
SymbolsRestrictSystemRootAccess	True, False	False	Determines if searching for .pdb files is allowed in the system root directory.	<SymbolsRestrictSystemRootAccess>True</SymbolsRestrictSystemRootAccess>

Docs: https://github.com/microsoft/codecoverage/blob/main/docs/configuration.md