coverlet.collector producing empty coverage file in CI only

[YairHalberstadt]

I'm experimenting with using coverlet for stronginject.
This is fine locally, but in the CI the coverlet file is empty:

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="1" branch-rate="1" version="1.9" timestamp="1613482081" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
  <sources />
  <packages />
</coverage>

Deterministic build is enabled, but we're using .Net 5, which I think should just automatically work according to your documentation.

reproducing

This is running in github actions on one of my branches.
You can see the github action results here: https://github.com/YairHalberstadt/stronginject/runs/1910881809
In the "Move Tests Coverage Reports" I cat one of the files, so you can see the coverage report:

You can view the workflow file here: https://github.com/YairHalberstadt/stronginject/actions/runs/571715677/workflow

Any help would be much appreciated, thanks!

[petli]

Is it only StrongInject.Tests.Unit that produces an empty coverage? The codacy step only logs an error about that one and not the others: https://github.com/YairHalberstadt/stronginject/runs/1910881809#step:10:43

[YairHalberstadt]

StrongInject.Tests.Unit is empty
StrongInject.Tests.Integration is empty
StrongInject.Extensions.DependencyInjection.Tests has results

Locally they all have results

[petli]

That one is weird. Can you try adding diagnostic logging and share the resulting file? Description on how to do that with coverlet.collectors here: https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#collectors-integration

[YairHalberstadt]

Sure will do.

Really appreciate your assistance!

By the way if it's easier for you than going backwards and forwards, feel free to modify the GitHub actions yourself and open a PR to trigger CI.

Obviously happy to do this all myself, so only if you actually prefer doing it yourself

[petli]

It's easier if you try it in your branch, so please go ahead :)

[YairHalberstadt]

You can see the diag file here: https://github.com/YairHalberstadt/stronginject/runs/1927012701

under "Move Tests Coverage Reports" where I cat it.

Would copy and paste it here, but it's huuuge

[petli]

Thanks, not much useful in log.txt, alas. https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md says there might be more files with more coverlet info in them, so maybe a ls log*.txt and cat log*.txt in the script will help print them.

[YairHalberstadt]

Ok, here's a gist with all coverlet logs files just for StrongInject.Tests.Unit: https://gist.github.com/YairHalberstadt/fec5b93708c8d773595a08691ca376a8

I can't tell you where one ends and the next begins, as I just did cat log* I'm afraid.

The test run is here: https://github.com/YairHalberstadt/stronginject/runs/1933761369?check_suite_focus=true

EDIT:

Note the display in github is truncated. Go here to see the full file

[petli]

The only issues I see there are related to xunit and the testhost, and I don't think that's something that can cause an issue with the coverage:

2021-02-19T08:18:37.1697976Z TpTrace Information: 0 : 2728, 1, 2021/02/19, 08:18:23.342, 133496827155, datacollector.dll, [coverlet]CoverletCoverageDataCollector: TestModules: '/home/runner/work/stronginject/stronginject/StrongInject.Tests.Unit/bin/Debug/net5.0/StrongInject.Tests.Unit.dll'
2021-02-19T08:18:37.1704414Z TpTrace Verbose: 0 : 2728, 1, 2021/02/19, 08:18:23.344, 133498997869, datacollector.dll, [coverlet]CoverletCoverageDataCollector: Initializing coverlet process with settings: "TestModule: '/home/runner/work/stronginject/stronginject/StrongInject.Tests.Unit/bin/Debug/net5.0/StrongInject.Tests.Unit.dll', IncludeFilters: '', IncludeDirectories: '', ExcludeFilters: '[coverlet.*]*', ExcludeSourceFiles: '', ExcludeAttributes: '', MergeWith: '', UseSourceLink: 'False'SingleHit: 'False'IncludeTestAssembly: 'False'SkipAutoProps: 'False'DoesNotReturnAttributes: ''"
2021-02-19T08:18:37.1708715Z TpTrace Verbose: 0 : 2728, 1, 2021/02/19, 08:18:23.378, 133532048912, datacollector.dll, [coverlet]Excluded module filter '[coverlet.*]*'
2021-02-19T08:18:37.1710657Z TpTrace Information: 0 : 2728, 1, 2021/02/19, 08:18:23.382, 133536950570, datacollector.dll, AssemblyResolver.OnResolve: Microsoft.Extensions.FileSystemGlobbing: Resolving assembly.
2021-02-19T08:18:37.1714295Z TpTrace Information: 0 : 2728, 1, 2021/02/19, 08:18:23.383, 133537260887, datacollector.dll, AssemblyResolver.OnResolve: Resolved assembly: Microsoft.Extensions.FileSystemGlobbing, from path: /home/runner/.nuget/packages/coverlet.collector/3.0.2/build/netstandard1.0/Microsoft.Extensions.FileSystemGlobbing.dll
2021-02-19T08:18:37.1718780Z TpTrace Verbose: 0 : 2728, 1, 2021/02/19, 08:18:23.400, 133554635303, datacollector.dll, [coverlet]Unable to instrument module: /home/runner/work/stronginject/stronginject/StrongInject.Tests.Unit/bin/Debug/net5.0/xunit.runner.utility.netcoreapp10.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.runner.utility\Extensions\DisposableExtensions.cs]
2021-02-19T08:18:37.1722570Z TpTrace Verbose: 0 : 2728, 1, 2021/02/19, 08:18:23.401, 133555076826, datacollector.dll, [coverlet]Unable to instrument module: /home/runner/work/stronginject/stronginject/StrongInject.Tests.Unit/bin/Debug/net5.0/xunit.core.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\common\ExceptionExtensions.cs]
2021-02-19T08:18:37.1726348Z TpTrace Verbose: 0 : 2728, 1, 2021/02/19, 08:18:23.401, 133555999375, datacollector.dll, [coverlet]Unable to instrument module: /home/runner/work/stronginject/stronginject/StrongInject.Tests.Unit/bin/Debug/net5.0/xunit.execution.dotnet.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.execution\Extensions\ReflectionAbstractionExtensions.cs]
2021-02-19T08:18:37.1730025Z TpTrace Verbose: 0 : 2728, 1, 2021/02/19, 08:18:23.402, 133556930624, datacollector.dll, [coverlet]Unable to instrument module: /home/runner/work/stronginject/stronginject/StrongInject.Tests.Unit/bin/Debug/net5.0/xunit.assert.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.assert\Asserts\Assert.cs]
2021-02-19T08:18:37.1735014Z TpTrace Verbose: 0 : 2728, 1, 2021/02/19, 08:18:23.403, 133557514855, datacollector.dll, [coverlet]Unable to instrument module: /home/runner/work/stronginject/stronginject/StrongInject.Tests.Unit/bin/Debug/net5.0/xunit.runner.visualstudio.dotnetcore.testadapter.dll, embedded pdb without local source files, [/_/src/xunit.runner.visualstudio/obj/Release/netcoreapp2.1/xunit.runner.visualstudio.dotnetcore.testadapter.Version.cs]
2021-02-19T08:18:37.1742017Z TpTrace Verbose: 0 : 2728, 1, 2021/02/19, 08:18:23.404, 133558111086, datacollector.dll, [coverlet]Unable to instrument module: /home/runner/work/stronginject/stronginject/StrongInject.Tests.Unit/bin/Debug/net5.0/xunit.runner.reporters.netcoreapp10.dll, embedded pdb without local source files, [C:\Dev\xunit\xunit\src\xunit.runner.reporters\Utility\EnvironmentHelper.cs]

[petli]

(EDIT: updated after reading up on source generators.)

I only have one other thought, at this point: since StrongInject.Extensions.DependencyInjection.Tests is the only one that produces a coverage result, there is a rather interesting difference between that and the other test projects.

The Unit and Integration test projects include the source generator only:

	<ProjectReference Include="..\StrongInject\StrongInject.csproj" OutputItemType="Analyzer">
		<SetTargetFramework>TargetFramework=netstandard2.0</SetTargetFramework>
	</ProjectReference>

The StrongInject.Extension.DependencyInjection.Test.csproj use that, but also a regular project reference to the extension package:

	<ProjectReference Include="..\StrongInject.Extensions.DependencyInjection\StrongInject.Extensions.DependencyInjection.csproj" />
	<ProjectReference Include="..\StrongInject\StrongInject.csproj" OutputItemType="Analyzer">
		<SetTargetFramework>TargetFramework=netstandard2.0</SetTargetFramework>
	</ProjectReference>

StrongInject.Extensions.DependencyInjection.csproj in turn references StrongInject\StrongInject.csproj, but as a regular ProjectReference.

Now I'm just guessing, but I reckon the coverage that the last test picks up is through this transitive reference, while there's something with the source generator reference that doesn't work in the CI build but does locally.

[petli]

I see that your source generator produces file with the suffix .generated.cs, and there was an issue recently about this that concluded that the recommended suffix is .g.cs, which is what coverlet checks for from 3.0.3: #1084 (comment)

It seems far-fetched that could cause a problem in CI only and not locally, but maybe worth a shot?

[YairHalberstadt]

@petli

Sorry I haven't tried your previous suggestion - I've been very busy and haven't got round to it. I'll give both your ideas a spin, hopefully soon.

Thanks for all your help!

[YairHalberstadt]

@petli

It looks like the problem is that StrongInject multitargets .Net Standard 2.0, and .Net Standard 2.1. The generator part is only included in the .Net Standard 2.0 target, and so my unit tests reference the .Net Standard 2.0 target.

I have no idea why that should only affect CI build, and not normal build.

[MarcoRossignoli]

Closing for stale conversation, feel free to reopen if needed.