Please document the IPC handshake mechanisms and open up the classes under Microsoft.Testing.Platform.IPC

[bradwilson]

The protocol documentation is a good start, but it leaves a lot of unanswered questions about how runners are intended to actually communicate with tests. Right now that information appears to be locked up inside of Microsoft.Testing.Extensions.VSTestBridge (for which I've been unable to find the official source). Reverse engineering the dotnet test interactions seems to indicate that it uses a command line switch (--internal-msbuild-node) that's expected to let the test know that it should open a named pipe for communication. I have not yet tried to determine how Test Explorer (in VS or in VS Code) works (or doesn't yet work) here.

This leaves me with two problems.

Understanding the expectations on unit test projects

I do not intend to use the VSTest bridge in xUnit.net v3, so I'm left trying to decode the interactions and support them myself. What are the command line switches that I'm expected to recognize and handle? How can I directly create and use an instance of NamedPipeClient? I'm currently hacking in using TestApplication but I'd like to understand how I can do what that does myself, as the API there isn't necessarily a good fit for me.

Understanding the expectations on unit test runners

In the process of supporting meta-runners like xunit.v3.console, which can run multiple test assemblies, I would like to become the server side of the IPC protocol. Given that there is no abstraction equivalent to TestApplication, how can I directly create and use an instance of NamedPipeServer to be the test host?

[MarcoRossignoli]

I do not intend to use the VSTest bridge in xUnit.net v3, so I'm left trying to decode the interactions and support them myself. What are the command line switches that I'm expected to recognize and handle? How can I directly create and use an instance of NamedPipeClient? I'm currently hacking in using TestApplication but I'd like to understand how I can do what that does myself, as the API there isn't necessarily a good fit for me.

You should not use the named pipe directly, at the moment the IPC namespace is used internally for quick communication between processes to support out-of-process extension for instance codecoverage or hang dump, where the platform will restart itself under "observation" by the out out of process extensions.

i.e.

mytest.exe pid 1("test controller" contains codecoverage, hang dump etc...)
         -> mytest.exe pid 2 ("test host" where adapter run tests)

The current internal pipeline infra is also used to have a "structured" communication between the msbuild node and the test application when we run under "dotnet test" using the current msbuild infrastructure and shipped in https://www.nuget.org/packages/Microsoft.Testing.Platform.MSBuild

The Microsoft.Testing.Platform.MSBuild is the testing platform MSBuild support and allows:

• integration with the current dotnet test infrastructure
• autogenerate the entry point using the MSBuild static extensions registration.
• move to the bin and interact with the new configuration file (a json).
• in future it could generate the launcher file used by IDEs(TBD)

This is the reason why the option starts with --internal-, options that start with it are hidden internal.

At the moment we have a "draft" for the json/tcp protocol and the implementation is built-in inside the platform and it's enabled using the --server and @drognanar can help with it.

[bradwilson]

How will the replacement dotnet test know that it's interacting with a test project that wants --server vs. --internal-msbuild-node?

And please don't forget that I am interested in being the test runner as well. I would like xunit.v3.console (and any third party runners written against xunit.v3.runner.utility) to be able to run v3 tests via the same IPC that is used for the new testing platform if possible, since there is negative value (on my personal time 😂) in maintaining two nearly identical IPC mechanisms.

[MarcoRossignoli]

How will the replacement dotnet test know that it's interacting with a test project that wants --server vs. --internal-msbuild-node?

Nuget package https://nuget.info/packages/Microsoft.Testing.Platform.MSBuild ships a MSBuild task/props/targets that integrates with the "current" dotnet test infra and start the testing application passing this "internal" argument for "structured" communication to have "better" UX and avoid overlapping.

I would not take any dependency at the moment on IPC for dotnet test. The protocol at the moment is internal and also we're planning a big change to dotnet test to have a UX that's similar the dotnet vstest but "lighter" that will use most probably pipes and custom protocol.
To be supported there you will have to take dependency on the new core platform because we don't have plan to share an "open protocol" for it for the moment, it's too early; or you have to have custom implementation of the MSBuild infra like we did with the package above.

The only current "draft" for a public protocol we have at the moment is the TCP/json-rpc one used by VS/VS-Code, but it's not complete enough to support all the features we need to have a "perfect" dotnet test experience. We will eventually replace this internal protocol with the public documented one when it(the tcp one) will be "complete".

cc: @drognanar @Evangelink

[bradwilson]

Okay, sounds like I should just wait until things are documented and public classes are available for both sides of the IPC mechanism.

[bradwilson]

Do you have a timeline for when this will be usable?

[MarcoRossignoli]

Do you have a timeline for when this will be usable?

I'll let you know soon cc: @pavelhorak @Evangelink

[MarcoRossignoli]

Do you have a timeline for when this will be usable?

We don't plan for the moment to expose an IPC public protocol if you want you can experiment with the current tcp/json "draft" we have published that is supported in VS/VS-Code in preview for which we don't have yet an ETA to release as stable.

cc: @pavelhorak @Evangelink @drognanar

[bradwilson]

We don't plan for the moment to expose an IPC public protocol

Then is your only supported usage model via the (currently closed source) VSTest adapter?

[bradwilson]

Also, is this the latest documentation?

https://github.com/microsoft/testfx/blob/main/docs/mstest-runner-protocol/001-protocol-intro.md#test-node

Or is there something newer? There are clearly nowhere near enough details in this one document to implement this without decompiling your implementation(s).

[MarcoRossignoli]

Or is there something newer? There are clearly nowhere near enough details in this one document to implement this without decompiling your implementation(s).

For what I know yes but I'm not the owner of this part so I'll @drognanar to drive you there.

@drognanar can you help @bradwilson with the current protocol draft?

[bradwilson]

Ping @drognanar

[bradwilson]

Ping @drognanar

[drognanar]

We currently have not exposed ObjectModel the way that we did for TPv2, so if you are not planning on using the TestApplication API, you'd need to create your own JSON-RPC server and parse the messages as specified in the document.

VS startup specifies the following flags: ./process.exe --server --client-port %PORT%. This then expects that you'll create a TpcClient that points to %PORT% and communicates over JSON-RPC.

The code that implements the VS -> test runner server mode is implemented around the following classes:

• https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/ServerMode/StreamMessageHandler.cs - how the TCP stream is encoded (as outlined by https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#baseProtocol)
• https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/ServerMode/FormatterUtilities.cs - entry point to how we serialize/deserialize the messages (converts our struct types to JSON objects as outlined in the protocol spec)
• https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/Hosts/ServerTestHost.cs - the main class that establishes the tcp client/establishes the json-rpc server/handles inbound and outbound messages

[drognanar]

You can also use System.Text.Json and vs-streamjsonrpc and avoid some of the boilerplate that we have, however in order to support native-aot and in order to support a platform that has no dependencies on other nuget packages, we had to make these utility classes ourselves.

[bradwilson]

This is the reason why the option starts with --internal-, options that start with it are hidden internal.

At the moment we have a "draft" for the json/tcp protocol and the implementation is built-in inside the platform and it's enabled using the --server and @drognanar can help with it.

@drognanar @MarcoRossignoli Is this statement still true? I'm working against 1.3.1 and I still see MSBuild passing --internal-msbuild-node. Is this ever going away?

For the record, my delineation in my newly injected entrypoint looks like this:

[global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
public class AutoGeneratedEntryPoint
{
    public static int Main(string[] args)
    {
        if (args.Length > 0 && (args[0] == "--internal-msbuild-node" || args[0] == "--server"))
            return global::Xunit.Runner.TestingPlatform.TestPlatformTestFramework.RunAsync(args).GetAwaiter().GetResult();
        else
            return global::Xunit.Runner.InProc.SystemConsole.ConsoleRunner.Run(args).GetAwaiter().GetResult();
    }
}

Does this seem correct to you?

(The TestPlatformTestFramework class is xUnit.net code that sets up the TestApplication via the standard CreateBuilderAsync, test framework registration, BuildAsync, RunAsync, etc.)

[bradwilson]

Also, do I need to add any other packages to make sure both dotnet test and Test Explorer work, other than Microsoft.Testing.Platform and Microsoft.Testing.Platform.MSBuild?

[bradwilson]

How can I verify whether Test Explorer is using the TPLv2 or TPLvNext APIs? It's easy to see when dotnet test is using the new stuff because the console output is quite different.

[bradwilson]

Would it be possible to update the code sample so that it works with Test Explorer? That would certainly help me (and others) understand what's involved in making that work.

[bradwilson]

I have questions regarding test case uniqueness: https://github.com/microsoft/vstest/blob/main/docs/RFCs/0017-Managed-TestCase-Properties.md

Tests Returning Multiple Results

There are some situations where the number of test cases cannot be determined at discovery time. In these cases, when the tests are executed multiple results are returned for a single discovered test case.

Uniqueness

The combination of ManagedType and ManagedMethod will be unique to a particular method within an assembly, but are not unique to a test case. This is due to the fact that when tests are data-driven, there can be many test cases that are executed by the same method. The test case ID is expected to be unique for every test case within an assembly. The test case ID should also be deterministic with respect to the method and its arguments. In other words, the test ID should not change given a particular method and a set of argument values (if any).

If I'm reading this correctly, then in the situation where I discover a single test case which will end up yielding multiple test results, then there is a conflict between the unique ID that is presented during discovery and the unique ID that would be used for reporting the TestNode; that is: the discovered test case unique ID is the unique ID of the test case, but the reported unique ID for the TestNode is the unique ID of the test (otherwise I'm reporting multiple results against the same unique ID, which appears to be illegal based on the Uniqueness section).

How does VSTest/Test Explorer reconcile this?

[bradwilson]

I'm going to open some new issues rather than adding more comments to this one.