Test explorer does not display tests after update

[cmorris98]

After the last update on 9-11 I get an error popup from the bottom right of visual studio code that says "container[title].push is not a function". This happens every time I press the refresh button. Below is a log output of the test discovery which all seems fine.

Starting extension Finding projects for pattern /Users/cmorris/Dev/Science/UnitTestsNetCore/UnitTests.csproj Found 1 matches for pattern Evaluating match /Users/cmorris/Dev/Science/UnitTestsNetCore/UnitTests.csproj Adding directory /Users/cmorris/Dev/Science/UnitTestsNetCore Executing dotnet test -t -v=q in /Users/cmorris/Dev/Science/UnitTestsNetCore Executing dotnet vstest "/Users/cmorris/Dev/Science/UnitTestsNetCore/bin/Debug/netcoreapp2.1/UnitTests.dll" /ListFullyQualifiedTests /ListTestsTargetPath:"/var/folders/_r/7hzckl0x1f9_93ywbm_sx4c40000gq/T/test-explorer-discover-v8ULrG/output.txt" in /Users/cmorris/Dev/Science/UnitTestsNetCore Discoverd tests with nunit. Found 3334 in 1 directories

Extension Info: .Net Core Test Explorer 0.5.3 (2018-09-11)
VSCode Info: Version 1.27.2 (1.27.2)

[stefanforsberg]

We did some changes in this area but we couldn't find a test case that caused this error. I realize this is a bit of a pita given the amount of tests you seem to have in your test project, but if you could narrow it down to which test or tests causes the error it would be extremely helpfull.

[cmorris98]

Is there a log file somewhere that will tell me? Or do I need to remove one at a time until it works?

[stefanforsberg]

I'm afraid the log won't tell you which one(s) it failed on so yes, it won't be pretty to find it. I'd help but I'm guessing this is not test data that you'd like to share with anyone.

[cmorris98]

yeah unfortunately its not a public repo and I cannot share the code base. I will try to narrow it down further early next week and get back to you. Thanks for the help.

[stefanforsberg]

My first suspects would be tests using TestFixtures and TestCase attributes

[TrevorVonSeggern]

I'm having the same error. My repo is public, if it helps here it is.

[stefanforsberg]

@TrevorVonSeggern

After some digging it seems like nunit reports it's test result in a slightly weird way, at least for how we operate in this extension.

When we do a test discovery we find the following two tests

1. NoData.Tests.GraphTests.GraphTests.Graph_Edges_SingleExists_Success(NoData.Tests.SharedExampleClasses.Dto,NoData.Tests.SharedExampleClasses.DtoChild,"favorite",False)
2. NoData.Tests.GraphTests.GraphTests.Graph_VertexContainsTypes_Success

When we run the actually tests they are reported back in the following format

<TestMethod codeBase="" adapterTypeName="executor://nunit3testexecutor/" className="NoData.Tests.GraphTests.GraphTests.Graph_Edges_SingleExists_Success(NoData.Tests.SharedExampleClasses.DtoChild,NoData.Tests.SharedExampleClasses" name="Graph_Edges_SingleExists_Success(NoData.Tests.SharedExampleClasses.DtoChild,NoData.Tests.SharedExampleClasses.DtoGrandChild,&quot;favorite&quot;,False)" />

<TestMethod codeBase="" adapterTypeName="executor://nunit3testexecutor/" className="NoData.Tests.GraphTests.GraphTests" name="Graph_VertexContainsTypes_Success" />

We use the className + name attribute values from the TestMethod element to construct the FQN but as you can see the className of the test for "Graph_Edges_SingleExists_Success" seem to contain parts of the test case data which confuses our extension.

I did a quick test with a more simple test case method and this seems to work so it's not something that is a general issue with nunit and test cases

        [TestCase]
        [TestCase(1,1)]
        public void OneTime(int a, int b)
        {
            (a+b).ShouldBe(2);
        }    

 <TestMethod adapterTypeName="executor://nunit3testexecutor/" className="NunitTests.TestClass2" name="OneTime(1,1)" />

We should probably add some handling on our end so that we don't stop to work all together but the causing issue in this case might be a bug in the nunit adapter... maybe?

[dchw]

@stefanforsberg To add some color, I see the same error only after my test have been run. I am using xUnit. I am in the same boat as @cmorris98 - this code cannot be public, but it does make heavy use of theories and test fixtures.

[stefanforsberg]

@cmorris98 @TrevorVonSeggern @dchw

Looks like this is an actual bug in vstest (microsoft/vstest#1677). Our extension previously hid this error but now we don't anymore (to be able to support named test fixtures).

Until this fix in vstest is shipped what we can do is to not blow up the whole extension when this error occur. That will mean that these tests does not show up in the tree but the other ones will. I'm not sure that's a good enough state for you guys in the meantime but it's the best we can do.

[TrevorVonSeggern]

@stefanforsberg That's interesting why it only picked up two tests in my project (should be ~100). I might be switching to xunit soon :p

[TrevorVonSeggern]

So I switched to Xunit (currently on the develop branch). I'm still getting the same error. Perhaps it's something unrelated to the NUnit runner.

[stefanforsberg]

@TrevorVonSeggern We have a seperate bug with dot's in test names (#143) which should be fixed soonish. Do you have your xunit branch pushed somewhere? Would be great to test the fix for 143 there.

[TrevorVonSeggern]

@stefanforsberg yeah that code is on the develop brach.

[stefanforsberg]

With the fix for #143 that branch behaves a lot better, 137 test discovered and run.

While test cases with many params looks less then optimal, they are now at least run.

[stefanforsberg]

We've released a new version, curious if it solves any of the problem listed in this issue.

[cmorris98]

I just updated and my tests now appear and no error message. So it looks to be resolved for me. The release notes dont list this issue, but perhaps a different issue was responsible for fixing this?

[dchw]

Working here for me too!

[stefanforsberg]

@cmorris98 Yes, we changes so failure to parse one test does not break the tree. If you're still using nunit you will still hit the bug I mentioned (#145 (comment)) but it should just affect tests that suffers from that bug.

[TrevorVonSeggern]

Works for me as well, good work!

[stefanforsberg]

Sweet, I'm closing this issue then =)