MSTest discovery issues

[GustavEikaas]

Description

MSTest is harder than xUnit and nUnit when it comes to discovering tests because it just returns the Testname and not the namespace. This introduces naming conflicts.

Issue

• dotnet vstest -lt questions microsoft/testfx#3725

Possible solution

I might be able to hack together an F# script using the resource below. Using fsi I might be able to drastically improve the test discovery

• https://github.com/microsoft/vstest/blob/main/playground/TestPlatform.Playground/Program.cs#L145

[GustavEikaas]

Using the snippet linked above I got the fully qualified name of every mstest, I also get the Executor which reveals if its MStest, xunit or nunit. I also get a guid which would be nice for lookups to avoid string matching issues. All in all looks very promising

Gives me following properties

{
            "Id": "<GUID>",
            "FullyQualifiedName": "<namespace>",
            "DisplayName": "<namespace>",
            "ExecutorUri": "executor://xunit/VsTestRunner2/netcoreapp",
            "Source": "<dll_path>",
            "CodeFilePath": "<source_filepath>",
            "LineNumber": 22,
            "Properties": [<insignificant stuff>]
          }

Using this I can fix all mstest compability issues, and also implement the ref with filepath for tests without relying on roslyn

[GustavEikaas]

Downside: Seems like this requires Visual Studio Test Platform installed. Which might be fine but its yet another dependency

EDIT: a sufficient version of this is shipped with the dotnet sdk

[GustavEikaas]

#r "nuget: Microsoft.TestPlatform.TranslationLayer, 17.11.0"
#r "nuget: Microsoft.VisualStudio.TestPlatform, 14.0.0"
#r "nuget: MSTest.TestAdapter, 3.3.1"
#r "nuget: MSTest.TestFramework, 3.3.1"
#r "nuget: Newtonsoft.Json, 13.0.3"

open Microsoft.TestPlatform.VsTestConsole.TranslationLayer
open Microsoft.VisualStudio.TestPlatform.ObjectModel
open Microsoft.VisualStudio.TestPlatform.ObjectModel.Client
open System
open System.Collections.Generic
open Newtonsoft.Json

module TestDiscovery =

    type Test = { Id: Guid; Namespace: string; Name: string; FilePath: string; Linenumber: int }

    type PlaygroundTestDiscoveryHandler() =
        interface ITestDiscoveryEventsHandler2 with
          member _.HandleDiscoveredTests(discoveredTestCases: IEnumerable<TestCase>) =
              let testCases = Seq.toList discoveredTestCases
              if testCases |> List.isEmpty |> not then
                  let tests = testCases |> List.map (fun s -> { Id = s.Id; Namespace = s.FullyQualifiedName; Name = s.DisplayName; FilePath = s.CodeFilePath; Linenumber = s.LineNumber })
                  let json = JsonConvert.SerializeObject(tests, Formatting.Indented)
                  Console.WriteLine(json)
              else
                  ()
          member _.HandleDiscoveryComplete(_: DiscoveryCompleteEventArgs, _: IEnumerable<TestCase>): unit = 
              ()
          member _.HandleLogMessage(_: Logging.TestMessageLevel, _: string): unit = 
              ()
          member _.HandleRawMessage(_: string): unit = 
              ()


    type TestSessionHandler() =
      let mutable testSessionInfo: TestSessionInfo option = None

      interface ITestSessionEventsHandler with
        member _.HandleStartTestSessionComplete(eventArgs: StartTestSessionCompleteEventArgs) =
            testSessionInfo <- Some(eventArgs.TestSessionInfo)
        member _.HandleLogMessage(_: Logging.TestMessageLevel, _: string): unit = 
            ()
        member _.HandleRawMessage(_: string): unit = 
            ()
        member _.HandleStopTestSessionComplete(_: StopTestSessionCompleteEventArgs): unit = 
            ()

      member _.TestSessionInfo
        with get() = testSessionInfo
        and set(value) = testSessionInfo <- value
              
    let main(argv: string[]) =
      if argv.Length <> 2 then
        printfn "Usage: fsi script.fsx <vstest-console-path> <test-dll-path>"
        1
      else 
        let console = argv.[0]

        let sourceSettings = """
            <RunSettings>
            </RunSettings>
            """

        let sources = argv.[1..]

        let environmentVariables = Dictionary<string, string>()
        environmentVariables.Add("VSTEST_CONNECTION_TIMEOUT", "999")
        environmentVariables.Add("VSTEST_DEBUG_NOBP", "1")
        environmentVariables.Add("VSTEST_RUNNER_DEBUG_ATTACHVS", "0")
        environmentVariables.Add("VSTEST_HOST_DEBUG_ATTACHVS", "0")
        environmentVariables.Add("VSTEST_DATACOLLECTOR_DEBUG_ATTACHVS", "0")

        let options = TestPlatformOptions(CollectMetrics = true)

        let r = VsTestConsoleWrapper(console, ConsoleParameters(EnvironmentVariables = environmentVariables))
        let sessionHandler = TestSessionHandler()
        let discoveryHandler = PlaygroundTestDiscoveryHandler()
        let testSession = 
          match sessionHandler.TestSessionInfo with
          | Some info -> info
          | None -> 
              new TestSessionInfo()

        r.DiscoverTests(sources, sourceSettings, options, testSession, discoveryHandler)
        0

    main fsi.CommandLineArgs.[1..]

[Issafalcon]

Hi @GustavEikaas - Just came here after stumbling on this very nice solution to a problem I've been trying to solve on neotest-dotnet. If you don't mind, I'm going to borrow this bit of ingenuity in my neotest adapter. Naturally, I'll credit you with this work!

Also, great work on this plugin! It's something I've wanted to do for a while, so fantastic that something already exsits!

[GustavEikaas]

Hi,
Thats amazing, my main motivation for writing this plugin is to improve the dotnet-vim ecosystem so knock yourself out! 😄
Let me know if you need some help or want to discuss solutions, I havent used the TranslationLayer package as much yet but seems you can do a lot of really cool stuff with it!

[Issafalcon]

Excellent! 👌

I'm wondering why you went with F# FSI option. I assume it's because it can be run as a script while c# would have needed a full .Net project to be integrated into the plugin?

[GustavEikaas]

Yes pretty much, I also enjoy F#.
I needed something that shipped with the dotnet cli and with as little overhead as possible. It would probably work just as well with C# but there would be more code and more maintenance