Can Test Explorer match the Expecto folder hierarchy?

[charlesroddie]

Given tests in a folder structure:

[<Tests>]
let tests = 
  testList "ListA" [
    testList "ListAA" [
      testCase "CaseAA1" id
      testCase "CaseAA2" id
    ]
    testCase "CaseAB1" id
  ]

Is there a way to have Test Explorer match this?

Project, Namespace, Class isn't right

Project, Class isn't right either

@cmeeren in #364 was getting some correspondence, but the samples there are very constrained - every test is exactly 2 levels deep in the hierarchy.

Is there a spec for the Test Explorer behaviour, or should I figure out by trial and error what Expecto structures result in what outputs and then try to make a set of consrained types (FolderLevel1, FolderLevel2) and then try to generate an altered Expecto folder structure to get the desired Text Explorer behaviour?

[charlesroddie]

@baronfel I'm taking up your suggestion to try Expecto (via https://github.com/YoloDev/YoloDev.Expecto.TestSdk) and it's good so far but I'm stuck at this point.

[charlesroddie]

Even "Rectangular hierarchy", in which folders of degree 0 contain only tests, and folders of degree n contain folders of degree n-1, and the main degree is 3, doesn't work:

There seems to be a big problem connecting Expecto to Test Explorer. Test Explorer may be hacked together with strings but it definitely allows for deeper hierarchies than this.

Am I right in posting here rather than in https://github.com/YoloDev/YoloDev.Expecto.TestSdk ?

[olivercoad]

Hi.
It looks like you're using Visual Studio.

I don't use Visual Studio very much anymore, but back in the day I used to use adamchester/expecto-adapter. It looks like it's been archived since then (I guess people use YoloDev.Expecto.TestSdk for that now), but I remember getting the nesting in the VS Test Explorer to work fine. See screenshot in adamchester/expecto-adapter#47. You could try seeing if it still works but it might need an older version of VS.

Usually I just use the terminal for running tests now (ftestCase and ftestList are useful) so don't know much about YoloDev.Expecto.TestSdk, but I tried your example in VSCode to see what it's like with Ionide.

The nesting works fine for what you had in your "Rectangular hierarchy".

"Some other test" doesn't even show up in the right place though, it should be nested under "List".
Also, it only seems to identify testCases with a function defined in-line (CaseAAA1 shows up but CaseAAB1 doesn't). And of course, tests generated in a for loop don't show up either.

There seems to be an issue with it Ionide or VSCode though. When I try to run tests from the test explorer it just hangs indefinitely showing the spinner. Nothing shows up in any of the VSCode output logs.

The TestResults/Ionide.trx file is generated successfully and has all the expected tests and test names, so I suspect if it actually ran properly it might show all the tests and in the correct spots.

The output of dotnet test -t is exactly what I'd expect it to be, as is the Ionide.trx output file, so I'd guess the issue is probably in the interaction between YoloDev.Expecto.TestSdk and Visual Studio. (And the Ionide/VSCode issue is probably unrelated)

The following Tests are available:
    List.Some other test
    List.ListA.ListAA.ListAAA.CaseAAA1
    List.ListA.ListAA.ListAAA.CaseAAA2
    List.ListA.ListAA.ListAAB.CaseAAB1
    List.ListA.ListAA.ListAAB.CaseAAB2
    List.ListA.ListAB.ListABA-generated-list.CaseABA0
    List.ListA.ListAB.ListABA-generated-list.CaseABA1
    List.ListA.ListAB.ListABA-generated-list.CaseABA2
    List.ListA.ListAB.ListABA-generated-list.CaseABA3
    List.ListA.ListAB.ListABA-generated-list.CaseABA4
    List.ListA.ListAB.ListABB.CaseABB1
    List.ListA.ListAB.ListABB.CaseABB2
...

[baronfel]

Note that the VSCode feature is in its infancy, so I'd err on the side of a bug in our test discovery and execution. This issue should focus purely on the discovery and execution of expecto tests via the Yolodev adapter, which provides the bridge to the 'dotnet test' world that powers the CLI and VS experiences.

[olivercoad]

How does the VSCode feature discover and execute tests then (and where is it implemented?) if it's not with the yolodev adapter? I thought it runs something like dotnet test --logger "trx;Ionide.trx" to generate the report, meaning it would be using the same "'dotnet test' world" as VS.

But agreed, this issue should focus on the Yolodev adapter with VS

[baronfel]

The discovery happens via syntax analysis, which is way faster but not as accurate. The execution should happen via dotnet test, however. Can you raise an issue about any test execution issues on the Ionide VSCode repo?