Does vstest support running in-process?

[ViktorHofer]

This is a question therefore I removed the default issue template.

As I haven't been following what changed in vstest in the last 9 months I was wondering if VSTest already supports running in-proc. Repositories that are lower in the stack like dotnet/runtime want to avoid dependencies on the networking stack, i.e. communication over Sockets. The last time I spoke with @nohwnd, VSTest heavily depended on Sockets communication and there was not a way to force an in-process execution.

Running in-proc should also enable testing single file applications. AFAIK not being able to run in-proc is the remaining blocker for us in dotnet/runtime to adopt VSTest fully and avoid the unsupported xunit.console test runner.

[nohwnd]

It is still the same, we don't want to endorse in process execution, instead we should endorse configuring the transport between processes, to have more options than sockets and tcpip. Problem is, we don't have much motivation other than runtime to do it though :/

[agocke]

I've been looking at this for a bit -- I think the best way to support this might be a custom test host process. Main problem right now seems to be that this code path isn't very well tested. In fact, I don't think it's ever worked.

vstest/src/Microsoft.TestPlatform.TestHostProvider/Hosting/DotnetTestHostManager.cs

Line 314 in c8f881a

if (string.IsNullOrEmpty(testHostPath))

That line goes looking for testhost.dll if testHostExeFound is false (which it always is if useCustomDotnetHostPath is true), and testHostPath is empty, which it always is as well.

I'll see how far I can get.

Eventual goal is that tests are compiled into an exe, and then the hosting layer itself is also compiled into that exe.

[agocke]

Never mind, I misunderstood the docs here. I thought "host" meant a custom exe that could run tests -- instead it looks like it means a dotnet core host.

I think, ideally, the concept would be generalized to be completely language agnostic, and simply communicate over the RPC framework.

[MarcoRossignoli]

I think there're a lot of benefits in moving towards that direction.
We should discuss a bit more on it.

[agocke]

Happy to help out if necessary. Also, I think the biggest advantage is not just in testing for dotnet/runtime -- the new version of xunit, 3.0, doesn't support DLL-based tests at all. It uses exes exclusively. That likely means it will not be compatible with vstest until this feature is implemented.

[nohwnd]

Custom testhost is already a thing. The normal testhost you use, is using the same extension point you would use to implement a custom ITestRuntimeProvider and there are few implementations that take advantage of it. Most of those use the same core that we provide, e.g. UWP testhost, but you are free to bring your own and implement it in any language you like as long as you are able to process the json messages TP sends you over the selected transport. The only managed part you need to provide is the implementation of ITestRuntimeProvider, in an extension assembly that we load.

In theory you should not need more than that, as there are abstractions for both the communication channel, and serialization. But in practice the communication channel leaks the details of using tcpip and sockets in multiple places in vstest.console, and we don't have a way to query the runtime provider for supported transports.

(and similarly for json serialization, that could also be customizable, but is hardcoded in many places, and defined as the serialization of choice in the design documents for test platform)

Adding an additional transport is something I looked into in the past, I just could not decide what it should be. I was thinking IO, but that does not play well with having multiple testhosts talking over it.

[agocke]

@nohwnd Could you point me to docs on how you would write your own host? In the code above, it looks like the vstest code exits if testhost.dll doesn't exist.

[nohwnd]

The code above (DotnetTestHostManager) is implementing the extension point for running .NET tests.

https://github.com/microsoft/vstest-docs/blob/main/RFCs/0025-Test-Host-Runtime-Provider.md

So it exits when it cannot find a .NET testhost. If this particular runtime provider will be used is determined by public bool CanExecuteCurrentRunConfiguration(string runsettingsXml) on the same class.

This side only manages what process is started and few other pieces. The actual implementation of host is in the testhost project and crossPlatEngine project.

I don't think there is any documentation how to implement your own host and it is quite involved, at least for .NET. This is where testhost receives and responds to requests.

https://github.com/microsoft/vstest/blob/main/src/Microsoft.TestPlatform.CrossPlatEngine/EventHandlers/TestRequestHandler.cs#L306

The main messages are for:

• version check
• start discovery
• report discovery progress
• report discovery completion
• start test execution with given dlls
• start test execution with given set of pre-discovered tests
• report test execution progress, and report test execution result

But as I say, implementing your own testhost would not help. vstest.console will still force tcpip and sockets on you.

[nohwnd]

I took a quick stab on this, but I don't have an idea of a good transport to use. Is memory mapped file low level enough?

[ViktorHofer]

but I don't have an idea of a good transport to use. Is memory mapped file low level enough?

Why would you need a (network/IO) transport protocol at all? Can't all the work be done in a single process? xunit.console.dll does that and it works well for us.

[nohwnd]

Multiple reasons. If any test crashes the process you won't get any results from any tests. You can't collect hang and crash dumps.. You can parallelize multiple assemblies that have different tfms.

[MarcoRossignoli]

Why would you need a (network/IO) transport protocol at all? Can't all the work be done in a single process? xunit.console.dll does that and it works well for us.

We're experimenting on something that could help this scenario and maybe also others, we can sync as soon as will be ready.

cc: @vitek-karas @sharwell @Evangelink @pavelhorak

[agocke]

If the network protocol is well defined, I don't mind using that to connect to vstest. It would be nice if there were a reusable library i could reference for this, but not a blocker

[ViktorHofer]

If the network protocol is well defined, I don't mind using that to connect to vstest. It would be nice if there were a reusable library i could reference for this, but not a blocker

When I worked on migrating from the xunit.console runner to VSTest in the past, @jkotas expressed the necessity for us to not rely on the networking stack for runtime or libraries test. At that time, the Mono runtime was still bringing-up their networking stack implementation and even though that is completed now, I agree with Jan that we should not add such a dependency if not for a very strong reason.

[jkotas]

For runtime and core libraries tests, it is ok to depend on Console to "transport" the test results. It is where the current xunit.console is. Dependencies on anything higher up the stack make troubleshooting of bugs in core runtime and bring up of new platforms difficult.

[nohwnd]

@jkotas the quotes confuse me :) When you say "transport" do you mean communicating between processes using standard IO? Or do you mean that we should run in single process and just write the test results to screen?

I just don't want to invest time into developing something based on a misunderstanding :)

[jkotas]

I have two primary concerns:

• Be able to run .NET runtime tests and get meaningful results even when e.g. System.IO or System.Net libraries are broken in the runtime being tested.
• Be mindful of .NET runtime CI costs. Creating multiple processes, setting up communication channels, etc. adds up.

Runing in single process and just writing the test results to screen is a simple way to take care of both these concerns.

[ceztko]

I am having a native crash on linux when running tests using dotnet test run on a precompiled mstest assembly. This crash doesn't happen on Windows using the same command. I'm already having some success in collecting the crash dump with --blame-crash and retrieving a backtrace, but the debugging experience would be better if I could analyze the stdout of the crashing test host process or put breakpoints, but both tasks are more difficult because the test is not run in-process, as the OP is asking. I'm asking as well for a workaround to allow in-process testing, or any other strategy I could use here to improve the debugging experience.

[nohwnd]

Thanks for the feedback we hear it.

[nohwnd]

Lot of the ideas here were materialized as Testing.Platform, please follow that for more low-level test running. https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-intro?tabs=dotnetcli

[ceztko]

Lot of the ideas here were materialized as Testing.Platform, please follow that for more low-level test running.

What about the request of this issue? Is there already an option for in process testing?

[MarcoRossignoli]

What about the request of this issue? Is there already an option for in process testing?

The new testing platform is hostable in any process, you find the architecture here https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Index.md and a sample here https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Source/TestingPlatformExplorer/Program.cs#L15

To make the porting easier at the moment for MSTest we're generating automatically the Program.cs thanks to the integration with MSBuild at build-time(https://www.nuget.org/packages/Microsoft.Testing.Platform.MSBuild/)

[ViktorHofer]

Appreciate the status update. Any update or information to follow along for xunit?

[Evangelink]

Any update or information to follow along for xunit?

We have created an issue and a PR to xunit repo (see xunit/visualstudio.xunit#402) but the owner is not keen on moving it forward as they prefer to focus on v3. I suggest that you upvote the issue and/or engage some discussion with Brad.

[bradwilson]

Happy to help out if necessary. Also, I think the biggest advantage is not just in testing for dotnet/runtime -- the new version of xunit, 3.0, doesn't support DLL-based tests at all. It uses exes exclusively.

This is true, but doesn't have to be true.

NCrunch needed a hook to be able to run tests in process, so that's available if you want to use v3 and run the tests in process: https://github.com/xunit/xunit/blob/e98cd61f9d1ed581e88797e5b42427fbeaa092c2/src/xunit.v3.runner.utility/Frameworks/v3/Xunit3.cs#L287-L291

It would not be difficult to construct your own v3 runner using this flag. The hard part for most people would be ensuring all the dependency resolution is provided, but I'm pretty sure y'all are more than capable of that part. 😄