Code coverage stops working if assembly contains interfaces from source generators

[martincostello]

Originally posted as reactiveui/refit#1073

Describe the bug

In a project using Refit 6.0.8 and coverlet.msbuild 3.0.2, upon updating from Refit 5.2.4, code coverage is no longer being collected for the assembly containing Refit interfaces.

Using coverlet's troubleshooting guide here, I narrowed my investigation down to this line of interest:

[coverlet] Unable to instrument module: C:\Coding\Projects\MyProject\tests\MyProject.Tests\bin\debug\net5.0\MyProject.dll, pdb without local source files, [C:\Coding\Projects\MyProject\src\MyProject\InterfaceStubGenerator.Core\Refit.Generator.InterfaceStubGenerator\IMyApi.g.cs]

As there's no file on disk here, that sort of makes sense, but causes an issue as it breaks all code coverage for the project for the sake of the one interface.

I'm guessing this is something to do with the usage of source generators in Refit.

The failure then slipped through several CI jobs for Refit updates until it was noticed due to #1083.

Steps To Reproduce

Clone https://github.com/martincostello/alexa-london-travel and run ./build.ps1.

The build script will eventually output something like this:

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | NaN% | NaN%   | NaN%   |
+---------+------+--------+--------+

Example build: https://github.com/martincostello/alexa-london-travel/runs/1869767942?check_suite_focus=true#step:4:78

This seems to be affecting multiple repos I have using Refit, not just this one.

Here's another example: https://github.com/martincostello/alexa-london-travel-site/runs/1870035564?check_suite_focus=true#step:4:160

Expected behavior

Assemblies containing Refit interfaces are instrumented correctly.

For the referenced repository, it should output something like this from build.ps1:

+--------------------+-------+--------+--------+
| Module             | Line  | Branch | Method |
+--------------------+-------+--------+--------+
| LondonTravel.Skill | 99.1% | 94.82% | 97.9%  |
+--------------------+-------+--------+--------+

+---------+-------+--------+--------+
|         | Line  | Branch | Method |
+---------+-------+--------+--------+
| Total   | 99.1% | 94.82% | 97.9%  |
+---------+-------+--------+--------+
| Average | 99.1% | 94.82% | 97.9%  |
+---------+-------+--------+--------+

Example build: https://github.com/martincostello/alexa-london-travel/runs/1869465349?check_suite_focus=true#step:4:80

[MarcoRossignoli]

Could be similar to #482 (comment)

Can you try to filter out that files with ExcludeByFile?

[martincostello]

I've tried a few different combinations, but it doesn't seem to work.

<ExcludeByFile>**/*.g.cs</ExcludeByFile>

From the .binlog:

[coverlet] Excluded source files filter '**/*.g.cs'
...
[coverlet] Unable to instrument module: C:\Coding\Projects\MyProject\tests\MyProject.Tests\bin\debug\net5.0\MyProject.dll, pdb without local source files, [C:\Coding\Projects\MyProject\src\MyProject\InterfaceStubGenerator.Core\Refit.Generator.InterfaceStubGenerator\Generated.g.cs]

[MarcoRossignoli]

ok thank you, I need to investigate.

[martincostello]

I've created a small repo here you can use to try and debug the issue: https://github.com/martincostello/coverlet-refit-repro

If you clone it and run build.ps1 it will output an msbuild.binlog file.

Below is screenshot showing the [coverlet] output.

On my laptop the two most relevant lines are:

[coverlet] Unable to instrument module: C:\Coding\martincostello\coverlet-refit-repro\Project2.Tests\bin\Debug\net5.0\Project1.dll, pdb without local source files, [C:\Coding\martincostello\coverlet-refit-repro\Project1\InterfaceStubGenerator.Core\Refit.Generator.InterfaceStubGenerator\IGitHub.g.cs]
[coverlet] Unable to instrument module: C:\Coding\martincostello\coverlet-refit-repro\Project2.Tests\bin\Debug\net5.0\Project2.dll, pdb without local source files, [C:\Coding\martincostello\coverlet-refit-repro\Project2\InterfaceStubGenerator.Core\Refit.Generator.InterfaceStubGenerator\Generated.g.cs]

[MarcoRossignoli]

Thanks a Martin!
You're amazing!
This weekend I'll tackle this issue, super busy week ☹️ until now.

[MarcoRossignoli]

@martincostello can you try with https://f.feedz.io/marcorossignoli/coverletunofficial/nuget/index.json version 3.0.3-preview.4 ?

[martincostello]

Sure - I'll take a look a bit later today. Thanks for digging into it 🙂

[martincostello]

Bingo. That works! 🥳

Repro Project

Before

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | NaN% | NaN%   | NaN%   |
+---------+------+--------+--------+

After

+----------+------+--------+--------+
| Module   | Line | Branch | Method |
+----------+------+--------+--------+
| Project1 | 100% | 100%   | 100%   |
+----------+------+--------+--------+
| Project2 | 100% | 100%   | 100%   |
+----------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | 100% | 100%   | 100%   |
+---------+------+--------+--------+

Internal Project

Before

+---------------------------+--------+--------+--------+
| Module                    | Line   | Branch | Method |
+---------------------------+--------+--------+--------+
| xxxxxxx.xxxxxxxxxx.Models | 97.47% | 90.9%  | 92.1%  |
+---------------------------+--------+--------+--------+
| xxxxxxxxxx.Views          | 100%   | 100%   | 100%   |
+---------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 97.81% | 92.59% | 92.85% |
+---------+--------+--------+--------+
| Average | 98.73% | 95.45% | 96.05% |
+---------+--------+--------+--------+

After

+---------------------------+--------+--------+--------+
| Module                    | Line   | Branch | Method |
+---------------------------+--------+--------+--------+
| xxxxxxx.xxxxxxxxxx.Models | 97.47% | 90.9%  | 92.1%  |
+---------------------------+--------+--------+--------+
| xxxxxxxxxx                | 88.55% | 74.07% | 95.7%  |
+---------------------------+--------+--------+--------+
| xxxxxxxxxx.Views          | 100%   | 100%   | 100%   |
+---------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 88.93% | 75.07% | 95.43% |
+---------+--------+--------+--------+
| Average | 95.33% | 88.32% | 95.93% |
+---------+--------+--------+--------+

[MarcoRossignoli]

Cool I'll release the fix asap

[markphillips100]

I've been seeing this issue very recently whilst attempting to implement source generators (and make use of them). I've created a sample demonstrating the issue here.

The sample is using coverlet.msbuild 3.0.2 in the test projects however I've also tried the 3.0.3-preview.4 with no success.

[martincostello]

I'm not sure what the general case fix is/should be, but the fix provided so far checks for files named to show they are generated.

By convention this is by the file's name ending in .g.cs. If you changed the name your generated source emits to follow this convention the sample should pick it up in the coverage.

[markphillips100]

So I'm not actually concerned whether the generated code is covered. I'm more concerned that the class/module consuming the generator is no longer covered due to the fact a source generator is being used, i.e. the consuming module doesn't show up in code coverage.

[martincostello]

Sure, but as in this case you control the generator, so my point was if you rename the emitted code's file name to end in .g.cs then the issue will resolve with the preview build of 3.0.3 as coverlet with ignore it and not trip up due to "missing local source files".

It may also be that you're unintentionally emitting the code into more projects that you intended, as happened in Refit which contributed to magnifying the impact of this bug in other projects of mine: reactiveui/refit#1075

[markphillips100]

Ah, I see your point now. Sorry. It hadn't occurred to me that coverlet was tripping up "because" of the file name. I'll change the demo and see if 3.0.3-preview.4 fixes it. Thanks @martincostello .

[markphillips100]

Your suggestion to change the source generated to use "source-filename.g" (the ".cs" is appended automatically by C# source generators) works using 3.0.3-preview.4 @martincostello . Thanks for your help.

I know you previously said a release would be made asap, is there a rough ETA. Not rushing you, just planning is all.

[martincostello]

Cool, glad that's unblocked you.

Rough ETA will be for Marco to answer.

[markphillips100]

Ha sorry @martincostello . I got confused between you and Marco. :-)

[MarcoRossignoli]

I got confused between you and Marco. :-)

Martin is now a coverlet maintainer specialized in "alpha testing" 😄 so it's ok you ask to him(joking)

I'm more concerned that the class/module consuming the generator is no longer covered due to the fact a source generator is being used

Coverlet skip by default all dlls doesn't have a corresponding local source files. In this way we don't need to ask to users to filter out not interesting code.

I know you previously said a release would be made asap, is there a rough ETA. Not rushing you, just planning is all.

This week end I'll release this fix, I'm trying to add also another update but I don't know if I'll make it.

[pankti11]

Will this also fix the coverlet.console? @MarcoRossignoli

[MarcoRossignoli]

Will this also fix the coverlet.console? @MarcoRossignoli

Yep the engine in shared between "drivers"

[MarcoRossignoli]

Just released 3.0.3 with fix

[martincostello]

Thanks Marco ❤️

[martincostello]

From .NET 6.0 preview 2 the .NET SDK uses source generators for compiling Razor views (link).

Just mentioning it here as it might over time it may cause more people to find this issue if they aren't running at least coverlet.msbuild 3.0.3.

[feliwir]

Hey, we're hitting a similar issue in https://github.com/OpenSAGE/OpenSAGE

I'm getting:

[coverlet] Unable to instrument module: /home/mbits/Development/OpenSAGE/src/OpenSage.Game.Tests/bin/Debug/net5.0/OpenSage.Game.dll, pdb without local source files, [/home/mbits/Development/OpenSAGE/src/OpenSage.Game/OpenSage.Game.CodeGen/OpenSage.Game.CodeGen.ScriptActionsGenerator/ScriptActions.Execution.generated.cs] (TaskId:181)

However we're already on version 3.0.3 of the NuGet. Any ideas?

[petli]

@feliwir coverlet expects a suffix of .g.cs, otherwise the file triggers the heuristics to ignore the project. See #1164 for a suggestion of a solution that loosens the heuristic a bit. An alternative is to add .generated.cs as another known suffix, or get OpenSAGE to adopt the .g.cs bit-of-a-convention.

[pepone]

@martincostello I hit a similar issue after start using the LoggerMessage source generator in .NET 6 preview, I already using 3.0.3 is this a known problem, is there a workaround?

[martincostello]

Is it an ASP.NET Core app that uses Razor? If so: #1140 (comment)

[pepone]

No just a .NET 6 assembly library, after start using the LoggerMessage source generator coverage output for NUnit tests is empty.

[martincostello]

You're probably hitting this issue then. The fix will be in either preview 6 or preview 7. dotnet/runtime#53275

[pepone]

@martincostello Thanks for the info