Migrate from vstest to Microsoft.Testing.Platform

[alexaka1]

Version

2.4.1

Platform

ubuntu-latest GH runner

Steps to reproduce

1. Setup an XUnit V3 project with Microsoft.Testing.Platform
2. Run dotnet test --logger GitHubActions -- --report-xunit-trx
3. No output is generated on GitHub, but a trx is created

Details

With the new Microsoft.Testing.Platform the --logger parameter is no longer parsed by dotnet test at all. Unfortunately there does not seem to be a migration path. The logging should be handled by the test framework according to the docs. Am I missing something here? Or is there a way to still use this logger with the new test platform? For xunit specifically, these are the parameters it supports via the new framework.

Checklist

• I have looked through existing issues to make sure that this bug has not been reported before
• I have provided a descriptive title for this issue
• I have made sure that this bug is reproducible on the latest version of the package
• I have provided all the information needed to reproduce this bug as efficiently as possible
• I have sponsored this project

[alexaka1]

For XUnit v3 specifically it looks like this is the only way to have custom reporters. It is likely that this library would have to be packaged for each test runner to support Microsoft.Testing.Platform. This makes sense, since dotnet test used VSTest as a runner, but testing frameworks have moved to their own runners over time, which Microsoft followed by deprecating VSTest, and now dotnet test exists in this weird limbo of just being a wrapper around N+1 test frameworks.

[Tyrrrz]

Removing the --logger abstraction is a very weird decision by Microsoft. The whole point of it was that you only need to implement the logger once and any testing framework could then report to it. Same thing with dotnet test.

I haven't been following the development of the new testing platform, what state is it in? I'm not at all keen on maintaining/distributing a separate logger for each framework.

[Tyrrrz]

I tried looking through the links you sent, as well as Googling around, and I still have no idea how a custom logger would look like with Microsoft.Testing.Platform. It seems that all of xUnit's related switches are for its own built-in loggers.

[Evangelink]

Hi @Tyrrrz,

Keep in mind that dotnet test is currently designed 1:1 for VSTest without any abstraction. Every argument before the -- is captured by dotnet test to be passed down to VSTest. At the moment, we are only injecting MTP (hacky) into VSTest workflow so we are limted on the experience we can provide. This will change for .NET 10 dotnet/sdk#45927.

As for the logging itself, it depends what you want to do. VSTest is mixing many concepts under logger (which is also conflicting with diagnostic) which is one of the things we broke/split into MTP where we now have loggers (designed for diagnostic) and reporters (designed for informative messages/reports). TRX is a reporter, the live terminal is a reporter... For the diagnostic logging, we have separate arguments, see https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli#options

---

EDIT

It seems that what you are looking for is to build a custom reporter. Please refer to this code sample: https://github.com/microsoft/testfx/tree/main/samples/public/mstest-runner/CustomReportExtension/CustomReportExtension and to this tech doc https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-architecture

EDIT 2
Happy to help, this week might be pretty busy for me but I can try to review/guide during evenings (CET)

[Tyrrrz]

@Evangelink thank you for the info.

Yes, I guess what I'm after is a custom reporter, according to the new terminology. Is there a list of all possible update messages and their properties?

Also, are there any downsides to providing the reporter/logger in the same package for both vstest/MTP? VStest would resolve the logger via conventions/reflection and for MTP the user would call an extension method on the builder instead. I really hate creating separate packages for each separate integration.

[Evangelink]

There is no good doc for the messages (I'll work on it). The main messages you may be interested into are:

• An attachment that is not linked to a test session or a test node (test case) https://github.com/microsoft/testfx/blob/46487a0c259530e7d74e7819df1fb53398bdd9c3/src/Platform/Microsoft.Testing.Platform/Messages/FileArtifacts.cs#L11

• An attachment that is linked to a particular test session (e.g. coverage, trx...) https://github.com/microsoft/testfx/blob/46487a0c259530e7d74e7819df1fb53398bdd9c3/src/Platform/Microsoft.Testing.Platform/Messages/FileArtifacts.cs#L67

• An attachment that is linked to a particular test node (test case) https://github.com/microsoft/testfx/blob/46487a0c259530e7d74e7819df1fb53398bdd9c3/src/Platform/Microsoft.Testing.Platform/Messages/FileArtifacts.cs#L128

• A change of state of a test node (test case) https://github.com/microsoft/testfx/blob/46487a0c259530e7d74e7819df1fb53398bdd9c3/src/Platform/Microsoft.Testing.Platform/Messages/TestNodeUpdateMessage.cs#L14

Also, are there any downsides to providing the reporter/logger in the same package for both vstest/MTP?

No.

the user would call an extension method on the builder instead

You can also provide an MSBuild hook (see https://github.com/microsoft/testfx/blob/main/src/Adapter/MSTest.TestAdapter/buildTransitive/common/MSTest.TestAdapter.props#L8-L16 and C# class linked https://github.com/microsoft/testfx/blob/main/src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/TestingPlatformBuilderHook.cs) that will be picked up by Microsoft.Testing.Platform.MSBuild and be emited as part of the SelfRegisteredExtensions class. We recommend that the hook simply calls the public extension method so that you ensure both have the same behavior.

[Tyrrrz]

Thanks @Evangelink.

I think I just need the equivalents of these events:

GitHubActionsTestLogger/GitHubActionsTestLogger/TestLogger.cs

Lines 17 to 19 in 4c26c3c

	events.TestRunStart += (_, args) => context.HandleTestRunStart(args);
	events.TestResult += (_, args) => context.HandleTestResult(args);
	events.TestRunComplete += (_, args) => context.HandleTestRunComplete(args);

So probably TestNodeUpdateMessage will do?

Also, what is the recommended approach to handing options for reporters with MTP? Since they're registered through code, I can imagine that passing options directly in the method call would work. However, I think some options are better off passed through command line since there will be different things that the user would want to configure on their CI environment.

Currently, the options are consumed through the --logger argument:

What's the idiomatic equivalent for doing the same with MTP? Just read Environment.CommandLine or something better?

[Evangelink]

You want to register a command line extension, example: https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Extensions.TrxReport/TrxCommandLine.cs

[Tyrrrz]

@Evangelink I see.

Looks like supporting MTP will be quite a fork, seeing as only about 20% of the code can be effectively shared. A lot of the documentation will also be different too.

I'd rather not maintain support for both MTP and vstest. Is the intent to deprecate vstest in the future? In which case, when is that future expected so I can migrate the logger/reporter entirely instead of supporting two paradigms?

[Evangelink]

Is the intent to deprecate vstest in the future?

Mine yes :) But quite frankly it will take a long time to be able to move away everyone (including big corps).

I'd be happy to provide the GH support directly under our repo as this seems like something we should be providing but I don't want to sound like I am stealing your idea/project. Happy to brainstorm with you what would be the best for you.

[Tyrrrz]

In a perfect scenario, I think I would migrate to using MTP exclusively and direct vstest users to older (now unmaintained) versions of the package. For this, I guess MTP would have to be stable enough and supported by all major testing frameworks.

Supporting both vstest and MTP at the same time would effectively mean maintaining two completely separate instances of the package. I'm not entirely against the idea, but that doesn't sound very exciting.

My reasoning for why there is not much of an overlap in the code between the two, is because most of the heavy lifting done by the library is interacting with vstest's events and extracting data from them.

I'll think on this for a bit and see what makes the most sense. In any case, supporting MTP seems like it would be a necessity given that the eventual goal is to make it the default option.

[Evangelink]

I was actually coming back to write that you could consider stopping support of VSTest given it's pretty stable and for sure nothing big will change there.

As for the support and stability of MTP. It's been released in v1 in Jan 2024 and we are currently in v1.6. For the support across various test frameworks, I have released a blogpost announcing the support for the major test frameworks https://devblogs.microsoft.com/dotnet/mtp-adoption-frameworks/

[Tyrrrz]

Okay, in that case that looks like the best path forward :)

I added this issue to my personal tracker, but can't promise when I'll be able to work on it. If someone wants to tackle this via PR, you're very welcome to.