[Microsoft.Testing.Extensions.CodeCoverage] `--coverage-settings` is not recognized when `PublishAot` is set

[sliekens]

Hi,

For some reason, I can't build my tests with Native AOT and then provide --coverage-settings to the MTP runner, why is that?

As a result, my CI started failing with exit code 5 after enabling Native AOT.

Actual behavior

dotnet test --configuration Release --coverage --coverage-settings coverage.settings

/__w/blablabla/SomeTests.dll (net10.0) Zero tests ran
Exit code: 5
  Standard output: Unknown option '--coverage-settings'

Expected behavior

Should work just the same with or without Native AOT.

Repro:

SomeTests.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <PublishAot>true</PublishAot>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" Version="18.1.0" />
    <PackageReference Include="TUnit" Version="1.2.0" />
  </ItemGroup>

</Project>

global.json

{
  "sdk": {
    "version": "10.0.100"
  },
  "test": {
    "runner": "Microsoft.Testing.Platform"
  }
}

coverage.settings

<?xml version="1.0" encoding="utf-8"?>
<Configuration>
  <Format>xml</Format>
  <DeterministicReport>True</DeterministicReport>
  <CodeCoverage>
    <AttributeFilters>
      <ExcludeAttributes>
        <Attribute>^System\.ObsoleteAttribute$</Attribute>
        <Attribute>^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$</Attribute>
      </ExcludeAttributes>
    </AttributeFilters>

    <Sources>
      <Exclude>
        <Source>.*\.generated\.cs</Source>
        <Source>.*\.g\.cs</Source>
      </Exclude>
    </Sources>

    <EnableStaticManagedInstrumentation>False</EnableStaticManagedInstrumentation>
    <EnableDynamicManagedInstrumentation>True</EnableDynamicManagedInstrumentation>
  </CodeCoverage>
</Configuration>

SomeTests.cs

public class SomeTests
{
    [Test]
    public async Task SampleTest()
    {
        bool result = true;
        await Assert.That(result).IsEqualTo(true);
    }
}

Output of dotnet test --help

When <PublishAot>false</PublishAot>

--coverage-settings is listed under Extension Options.

Description:
  .NET Test Command for Microsoft.Testing.Platform (opted-in via 'global.json' file). This only supports Microsoft.Testing.Platform and doesn't support VSTest. For more information, see https://aka.ms/dotnet-test.

Usage:
  dotnet test [options] [platform options] [extension options]

Options:
  --project <PROJECT_PATH>                        Defines the path of the project file to run (folder name or full path). If not specified, it defaults to the current directory.
  --solution <SOLUTION_PATH>                      Defines the path of the solution file to run. If not specified, it defaults to the current directory.
  --test-modules <EXPRESSION>                     Run tests for the specified test modules.
  --root-directory <ROOT_PATH>                    The test modules have the specified root directory.
  --results-directory <RESULTS_DIR>               The directory where the test results will be placed.
                                                  The specified directory will be created if it does not exist.
  --config-file <CONFIG_FILE>                     Specifies a testconfig.json file.
  --diagnostic-output-directory <DIAGNOSTIC_DIR>  Output directory of the diagnostic logging.
                                                  If not specified the file will be generated inside the default 'TestResults' directory.
  --max-parallel-test-modules <NUMBER>            The max number of test modules that can run in parallel.
  --minimum-expected-tests <NUMBER>               Specifies the minimum number of tests that are expected to run.
  -a, --arch <ARCH>                               The target architecture.
  -e, --environment <NAME="VALUE">                Sets the value of an environment variable. 
                                                  Creates the variable if it does not exist, overrides if it does. 
                                                  This argument can be specified multiple times to provide multiple variables.
  
                                                  Examples:
                                                  -e VARIABLE=abc
                                                  -e VARIABLE="value with spaces"
                                                  -e VARIABLE="value;seperated with;semicolons"
                                                  -e VAR1=abc -e VAR2=def -e VAR3=ghi
  -c, --configuration <CONFIGURATION>             The configuration to use for running tests. The default for most projects is 'Debug'.
  -f, --framework <FRAMEWORK>                     The target framework to run tests for. The target framework must also be specified in the project file.
  --os <OS>                                       The target operating system.
  -r, --runtime <RUNTIME_IDENTIFIER>              The target runtime to test for.
  -v, -verbosity <LEVEL>                          Set the MSBuild verbosity level. Allowed values are q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic].
  --no-restore                                    Do not restore the project before building. [default: False]
  --no-build                                      Do not build the project before testing. Implies --no-restore. [default: False]
  --no-ansi                                       Disable ANSI output. [default: False]
  --no-progress                                   Disable progress reporting. [default: False]
  --output <Detailed|Normal>                      Verbosity of test output.
  --list-tests <list-tests>                       List the discovered tests instead of running the tests.
  --no-launch-profile                             Do not attempt to use launchSettings.json or [app].run.json to configure the application. [default: False]
  --no-launch-profile-arguments                   Do not use arguments specified in launch profile to run the application. [default: False]
  -?, -h, --help                                  Show command line help.

Waiting for options and extensions...
Platform Options:
  --debug                         Allows to pause execution in order to attach to the process for debug purposes.
  --diagnostic                    Enable the diagnostic logging. The default log level is 'Trace'.
                                  The file will be written in the output directory with the name log_[yyMMddHHmmssfff].diag
  --diagnostic-file-prefix        Prefix for the log file name that will replace '[log]_.'
  --diagnostic-synchronous-write  Force the built-in file logger to write the log synchronously.
                                  Useful for scenario where you don't want to lose any log (i.e. in case of crash).
                                  Note that this is slowing down the test execution.
  --diagnostic-verbosity          Define the level of the verbosity for the --diagnostic.
                                  The available values are 'Trace', 'Debug', 'Information', 'Warning', 'Error', and 'Critical'.
  --exit-on-process-exit          Exit the test process if dependent process exits. PID must be provided.
  --filter-uid                    Provides a list of test node UIDs to filter by.
  --help                          Show the command line help.
  --ignore-exit-code              Do not report non successful exit value for specific exit codes
                                  (e.g. '--ignore-exit-code 8;9' ignore exit code 8 and 9 and will return 0 in these case)
  --info                          Display .NET test application information.
  --timeout                       A global test execution timeout.
                                  Takes one argument as string in the format <value>[h|m|s] where 'value' is float.

Extension Options:
  --adaptive-metrics        Enable detailed metrics logging for adaptive parallelism
  --coverage                Collect the code coverage using dotnet-coverage tool.
  --coverage-output         The output of the generated code coverage report.
                            If only filename is specified then the coverage report will be generated in the '--results-directory' directory.
  --coverage-output-format  Output file format. Supported values: 'coverage', 'xml' and 'cobertura'
  --coverage-settings       The path to the code coverage XML settings file.
  --detailed-stacktrace     Display TUnit internals within stack traces
  --disable-logo            Disables the TUnit logo when starting a test session
  --fail-fast               Cancel the test run after the first test failure
  --github-reporter-style   GitHub reporter output style: collapsible (default) or full
  --log-level               Minimum log level for test output: Trace, Debug, Information, Warning, Error, Critical, None (default: Information)
  --maximum-failed-tests    Specifies a maximum number of test failures that, when exceeded, will abort the test run.
  --maximum-parallel-tests  Override maximum parallel tests (defaults to auto-detected optimal value)
  --parallelism-strategy    Parallelism strategy: fixed or adaptive (default: adaptive)
  --reflection              Enable reflection mode for test discovery and execution (defaults to AOT-optimized mode)
  --report-trx              Enable generating TRX report
  --report-trx-filename     The name of the generated TRX report
  --test-parameter          Custom parameters to pass to TUnit
  --treenode-filter         Use a tree filter to filter down the tests to execute

When <PublishAot>true</PublishAot>

--coverage-settings is not listed anymore under Extension Options.

Description:
  .NET Test Command for Microsoft.Testing.Platform (opted-in via 'global.json' file). This only supports Microsoft.Testing.Platform and doesn't support VSTest. For more information, see https://aka.ms/dotnet-test.

Usage:
  dotnet test [options] [platform options] [extension options]

Options:
  --project <PROJECT_PATH>                        Defines the path of the project file to run (folder name or full path). If not specified, it defaults to the current directory.
  --solution <SOLUTION_PATH>                      Defines the path of the solution file to run. If not specified, it defaults to the current directory.
  --test-modules <EXPRESSION>                     Run tests for the specified test modules.
  --root-directory <ROOT_PATH>                    The test modules have the specified root directory.
  --results-directory <RESULTS_DIR>               The directory where the test results will be placed.
                                                  The specified directory will be created if it does not exist.
  --config-file <CONFIG_FILE>                     Specifies a testconfig.json file.
  --diagnostic-output-directory <DIAGNOSTIC_DIR>  Output directory of the diagnostic logging.
                                                  If not specified the file will be generated inside the default 'TestResults' directory.
  --max-parallel-test-modules <NUMBER>            The max number of test modules that can run in parallel.
  --minimum-expected-tests <NUMBER>               Specifies the minimum number of tests that are expected to run.
  -a, --arch <ARCH>                               The target architecture.
  -e, --environment <NAME="VALUE">                Sets the value of an environment variable. 
                                                  Creates the variable if it does not exist, overrides if it does. 
                                                  This argument can be specified multiple times to provide multiple variables.
  
                                                  Examples:
                                                  -e VARIABLE=abc
                                                  -e VARIABLE="value with spaces"
                                                  -e VARIABLE="value;seperated with;semicolons"
                                                  -e VAR1=abc -e VAR2=def -e VAR3=ghi
  -c, --configuration <CONFIGURATION>             The configuration to use for running tests. The default for most projects is 'Debug'.
  -f, --framework <FRAMEWORK>                     The target framework to run tests for. The target framework must also be specified in the project file.
  --os <OS>                                       The target operating system.
  -r, --runtime <RUNTIME_IDENTIFIER>              The target runtime to test for.
  -v, -verbosity <LEVEL>                          Set the MSBuild verbosity level. Allowed values are q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic].
  --no-restore                                    Do not restore the project before building. [default: False]
  --no-build                                      Do not build the project before testing. Implies --no-restore. [default: False]
  --no-ansi                                       Disable ANSI output. [default: False]
  --no-progress                                   Disable progress reporting. [default: False]
  --output <Detailed|Normal>                      Verbosity of test output.
  --list-tests <list-tests>                       List the discovered tests instead of running the tests.
  --no-launch-profile                             Do not attempt to use launchSettings.json or [app].run.json to configure the application. [default: False]
  --no-launch-profile-arguments                   Do not use arguments specified in launch profile to run the application. [default: False]
  -?, -h, --help                                  Show command line help.

Waiting for options and extensions...
Platform Options:
  --debug                         Allows to pause execution in order to attach to the process for debug purposes.
  --diagnostic                    Enable the diagnostic logging. The default log level is 'Trace'.
                                  The file will be written in the output directory with the name log_[yyMMddHHmmssfff].diag
  --diagnostic-file-prefix        Prefix for the log file name that will replace '[log]_.'
  --diagnostic-synchronous-write  Force the built-in file logger to write the log synchronously.
                                  Useful for scenario where you don't want to lose any log (i.e. in case of crash).
                                  Note that this is slowing down the test execution.
  --diagnostic-verbosity          Define the level of the verbosity for the --diagnostic.
                                  The available values are 'Trace', 'Debug', 'Information', 'Warning', 'Error', and 'Critical'.
  --exit-on-process-exit          Exit the test process if dependent process exits. PID must be provided.
  --filter-uid                    Provides a list of test node UIDs to filter by.
  --help                          Show the command line help.
  --ignore-exit-code              Do not report non successful exit value for specific exit codes
                                  (e.g. '--ignore-exit-code 8;9' ignore exit code 8 and 9 and will return 0 in these case)
  --info                          Display .NET test application information.
  --timeout                       A global test execution timeout.
                                  Takes one argument as string in the format <value>[h|m|s] where 'value' is float.

Extension Options:
  --adaptive-metrics        Enable detailed metrics logging for adaptive parallelism
  --coverage                Collect the code coverage using dotnet-coverage tool.
  --coverage-output         The output of the generated code coverage report.
                            If only filename is specified then the coverage report will be generated in the '--results-directory' directory.
  --coverage-output-format  Output file format. Supported values: 'coverage', 'xml' and 'cobertura'
  --detailed-stacktrace     Display TUnit internals within stack traces
  --disable-logo            Disables the TUnit logo when starting a test session
  --fail-fast               Cancel the test run after the first test failure
  --github-reporter-style   GitHub reporter output style: collapsible (default) or full
  --log-level               Minimum log level for test output: Trace, Debug, Information, Warning, Error, Critical, None (default: Information)
  --maximum-failed-tests    Specifies a maximum number of test failures that, when exceeded, will abort the test run.
  --maximum-parallel-tests  Override maximum parallel tests (defaults to auto-detected optimal value)
  --parallelism-strategy    Parallelism strategy: fixed or adaptive (default: adaptive)
  --reflection              Enable reflection mode for test discovery and execution (defaults to AOT-optimized mode)
  --report-trx              Enable generating TRX report
  --report-trx-filename     The name of the generated TRX report
  --test-parameter          Custom parameters to pass to TUnit
  --treenode-filter         Use a tree filter to filter down the tests to execute

[Youssef1313]

@fhnaseer

[sliekens]

@fhnaseer thanks for the link. Is this option specific for MSTest? I am using TUnit.

[fhnaseer]

I am looking into this. Meanwhile, you can try using testconfig.json as code coverage config can be provided in this as well.
https://github.com/microsoft/codecoverage/blob/main/docs/configuration.md#testconfigjson
https://github.com/microsoft/codecoverage/blob/main/docs/configuration.md#testconfigjson-file

[sliekens]

@fhnaseer thanks, unfortunately that doesn't fix it. My testconfig.json:

{
  "Configuration": {
    "CoverageFileName": "coverage.xml",
    "Format": "xml",
    "DeterministicReport": true,
    "CodeCoverage": {
      "Attributes": {
        "Exclude": [
          "^System\\.ObsoleteAttribute$",
          "^System\\.CodeDom\\.Compiler\\.GeneratedCodeAttribute$"
        ]
      },
      "Sources": {
        "Exclude": [
          ".*\\.generated\\.cs",
          ".*\\.g\\.cs"
        ]
      },
      "EnableStaticManagedInstrumentation": false,
      "EnableDynamicManagedInstrumentation": true
    }
  },
  "codeCoverage": {
    "Configuration": {
      "CoverageFileName": "coverage.xml",
      "Format": "xml",
      "DeterministicReport": true,
      "CodeCoverage": {
        "Attributes": {
          "Exclude": [
            "^System\\.ObsoleteAttribute$",
            "^System\\.CodeDom\\.Compiler\\.GeneratedCodeAttribute$"
          ]
        },
        "Sources": {
          "Exclude": [
            ".*\\.generated\\.cs",
            ".*\\.g\\.cs"
          ]
        },
        "EnableStaticManagedInstrumentation": false,
        "EnableDynamicManagedInstrumentation": true
      }
    }
  }
}

My test command:

dotnet test --framework net10.0 --config-file tests/testconfig.json --coverage /p:PublishAot=true

The result is a file like TestResults/a178983e-20d1-46b0-8cfb-f62a8ccb8693.coverage (binary format) instead of the expected coverage.xml (XML format).

Running the same test command with /p:PublishAot=false correctly produces coverage.xml.