0% coverage with different paths and SourceLink/Deterministic builds

[jborean93]

Describe the bug
I am trying to collect coverage for a C# assembly and while things work just fine locally I'm having difficulties getting it to work in CI as I'm building my assembly on one runner then running the tests on another runner using that assembly.

I've tried doing this just normally without any special setup but this doesn't work as expected on different hosts because the paths inside the pdb are different on Linux vs Windows. For example the base build https://github.com/jborean93/CoverletTesting/actions/runs/7080112125/job/19267536711 produces a PDB with

I could not find any configuration option to map paths to an equivalent alternative that I've seen in other coverage tools like python coverage paths provide.

The next thing I tried was to use SourceLink and deterministic builds. I created a POC for this in the PR branch jborean93/CoverletTesting#1. When inspecting the pdb generated for the net6.0 build I can see the SourceLink was configured with a deterministic path

But unfortunately both the Linux and Windows runners report 0% coverage for the assemblies built like this.

There is a reference that deterministic builds only work for msbuild and collectors at https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/DeterministicBuild.md but I was hoping whether to clarify

• if this also includes the global tool, if so why does it have --enable-source-link
• if there are plans on adding support for it in the future
• if there are any workarounds for collecting in a subprocess not through dotnet test

To Reproduce
I've created a very basic repo that reproduces the issue when run in it https://github.com/jborean93/CoverletTesting. It can also be run locally if you have PowerShell installed on Linux or manually rename the checkout after building it.

Expected behavior
Coverage is collected

Actual behavior
No coverage is collected, there is no error but coverlet is unable to find any coverage.

Configuration (please complete the following information):
Please provide more information on your .NET configuration:
* Which coverlet package and version was used?
* 6.0.0
* Which version of .NET is the code running on?
* net6.0 (compiled with net8.0)
* What OS and version, and what distro if applicable?
* All
* What is the architecture (x64, x86, ARM, ARM64)?
* Tested on x64
* Do you know whether it is specific to that configuration?
* I don't believe so but could be wrong

Additional context
This is for PowerShell binary modules which are compiled dotnet assemblies. I'm trying to ensure that what I'm testing against is the same assembly being built which is why I'm building it first on one host then testing. I could technically build and test in the same step but that feels wrong to me especially since it may not be the same assembly that I publish. Also using a deterministic build would be nice but I don't care about that as much.

❗ Please also read Known Issues

[Bertk]

Hi, please check the integration test for deterministic build within coverlet repository.

You will find dotnet build -c Debug /p:DeterministicSourcePaths=true" for the determisticbuild project which uses coverlet.msbuild and coverlet.collector for the specific test cases.

The target ReferencedPathMaps which is available using the Nuget packages will create the required "CoverletSourceRootsMapping_*" file.

  <Target Name="ReferencedPathMaps" BeforeTargets="CoreCompile" DependsOnTargets="ResolveProjectReferences" >
    <MSBuild Projects="@(AnnotatedProjects->'%(FullPath)')"
             Targets="$(_CoverletSourceRootTargetName)"
             Properties="TargetFramework=%(AnnotatedProjects.NearestTargetFramework)"
             SkipNonexistentTargets="true">
      <Output TaskParameter="TargetOutputs"
              ItemName="_LocalTopLevelSourceRoot" />
    </MSBuild>
    <ItemGroup>
      <_byProject Include="@(_LocalTopLevelSourceRoot->'%(MSBuildSourceProjectFile)')" OriginalPath="%(Identity)" />
      <_mapping Include="@(_byProject->'%(Identity)|%(OriginalPath)=%(MappedPath)')" />
    </ItemGroup>
    <PropertyGroup>
      <_sourceRootMappingFilePath>$([MSBuild]::EnsureTrailingSlash('$(OutputPath)'))CoverletSourceRootsMapping_$(AssemblyName)</_sourceRootMappingFilePath>
    </PropertyGroup>
    <WriteLinesToFile File="$(_sourceRootMappingFilePath)" Lines="@(_mapping)"
                      Overwrite="true" Encoding="Unicode"
                      Condition="'@(_mapping)'!=''"
                      WriteOnlyWhenDifferent="true" />
    <ItemGroup>
      <FileWrites Include="$(_sourceRootMappingFilePath)" Condition="'@(_mapping)'!=''" />
    </ItemGroup>
  </Target>

Please check also Documentation/DeterministicBuild.md

This scenario is not supported but maybe this information helps.

[jborean93]

Thanks for the suggestion unfortunately I was unable to get it producing a valid file as _byProfile and _mapping was not being populated. I tried manually creating the file based on https://github.com/coverlet-coverage/coverlet/blob/master/test/coverlet.core.tests/TestAssets/CoverletSourceRootsMappingTest but when looking at the code for coverlet.console it never looks up this path and sets the mapping.

The coverlet.console creates the SourceRootTranslator at

coverlet/src/coverlet.console/Program.cs

Line 163 in 21d3964

serviceCollection.AddSingleton<ISourceRootTranslator, SourceRootTranslator>(provider => new SourceRootTranslator(provider.GetRequiredService<ILogger>(), provider.GetRequiredService<IFileSystem>()));

This calls this constructor which sets an empty source translator

coverlet/src/coverlet.core/Helpers/SourceRootTranslator.cs

Lines 28 to 33 in 21d3964

	public SourceRootTranslator(ILogger logger, IFileSystem fileSystem)
	{
	_logger = logger ?? throw new ArgumentNullException(nameof(logger));
	_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
	_sourceRootMapping = new Dictionary<string, List<SourceRootMapping>>();
	}

I've opened a PR that adds a new option to specify this path mapping file through the console tool. I've not contributed to this repo before so I'm not 100% sure what the process would be but hopefully we can iterate on that and get something in. I've tested the changed locally and can confirm the path mapping allows me to capture the coverage for my assembly when it is built with a deterministic build and even with just a normal build where the paths changed.

[Bertk]

@jborean93 Thank you for the contribution. Could you please also add integration tests for this enhancement.

@MarcoRossignoli @daveMueller I do not see a blocking topic but I am not using coverlet.console .NET tool. Could you please send your feedback.

[jborean93]

Could you please also add integration tests for this enhancement.

Pushed a commit with an integration test, hopefully it's what you are looking for.

[Bertk]

@jborean93 Thank you for adding the integration test.

Is it possible to check the content of the generated coverage (Cobertura) file content like here
https://github.com/coverlet-coverage/coverlet/blob/21d3964f0399a8b353894806f774fdcf973b4324/test/coverlet.integration.tests/DeterministicBuild.cs#L65C1-L74C1

[jborean93]

Is there a purpose behind checking that? The report will still contain the actual paths of the files, this new option just allowed the pdb paths which are set to a deterministic build (or other path) to the local ones. I'm not aware of a way for the coverlet global tool to emit a report with deterministic paths in it.

[Bertk]

@jborean93 Fine with me. The GlobalTool.md description of PR already clarifies it.