XUnit inline data is lost after test is run

[stefanforsberg]

On test discovery

After tests have run

[GeorchW]

Uh, is this some issue with the new parsing? I'll have a look in the next few days I think

[stefanforsberg]

@GeorchW Hey, that is what I suspected at first but I tried it with the 0.7.1 release and it behaved in the same way. I didn't have a lot of time to troubleshoot over the weekend but will take a closer look this week.

[GeorchW]

Hmm, I can't reproduce this issue, even if I disable all other extensions.

[stefanforsberg]

Hmm, even if you open the xunit solution that is included in the repo (in the test folder)?

[GeorchW]

Yes, that's the one I've tried it with.

[stefanforsberg]

Hmm intriguing!

I'll give it another go.

[GeorchW]

I mean, the bug is probably somewhere here. I'm not sure why it makes sense to reset the discovered tests when changing the results.

Oh, and while I'm at it, this three lines below is in O(n²). The filtering should probably be done with a set, not an array. Generally, this entire "loop through all tests" for each folder and each item, also seen in testNode.ts, is really inefficient code.

[stefanforsberg]

@GeorchW I hear you but let's try to keep this thread about the original issue =)

When I execute this command dotnet test --no-build --logger "trx;LogFileName=C:\SomePath\0.trx" --filter "FullyQualifiedName=XunitTests.TestClass3.Fail" I end up with the following xml (the TestDefinition node)

<TestDefinitions>
    <UnitTest name="XunitTests.TestClass3.Fail(value: 2)" storage="d:\SomePath\vscode-dotnet-test-explorer\test\xunittests\bin\debug\netcoreapp3.1\xunittests.dll" id="5d23175e-8479-779f-39f4-23aa2608428b">
      <Execution id="ceaf151e-e91c-4f92-ba85-4af475de7c28" />
      <TestMethod codeBase="d:\SomePath\vscode-dotnet-test-explorer\test\xunittests\bin\Debug\netcoreapp3.1\XunitTests.dll" adapterTypeName="executor://xunit/VsTestRunner2/netcoreapp" className="XunitTests.TestClass3" name="Fail" />
    </UnitTest>
  </TestDefinitions>

Since we are fetching the classname and method name from here it makes sense to me why the tree looks like it does but I don't really understand how it works for you. In order to rule out the possibility of different sdk versions containing different versions of vstest would you mind executing the above command in your context where things work and see if the resulting xml differ?

[GeorchW]

Sure, this is the part for me:

<TestDefinitions>
  <UnitTest name="XunitTests.TestClass3.Fail(value: 2)" storage="/path/to/the/repo/vscode-dotnet-test-explorer/test/xunittests/bin/debug/netcoreapp3.1/xunittests.dll" id="5d23175e-8479-779f-39f4-23aa2608428b">
    <Execution id="285b15ac-175b-4204-8d35-32dab901e73d" />
    <TestMethod codeBase="/path/to/the/repo/vscode-dotnet-test-explorer/test/xunittests/bin/Debug/netcoreapp3.1/XunitTests.dll" adapterTypeName="executor://xunit/VsTestRunner2/netcoreapp" className="XunitTests.TestClass3" name="XunitTests.TestClass3.Fail(value: 2)" />
  </UnitTest>
</TestDefinitions>

Looks pretty similar to me... EDIT: nvm, I'm an idiot

For context:

$ dotnet --version
3.1.103

[stefanforsberg]

Ah ok, then we need to figure out why the trx-files are different. I’ll continue investigating tomorrow :)

[stefanforsberg]

For me

dotnet --version
3.1.300

If I downgrade my sdk version to the one you are running I get the expected behavior. So something has changed in the way the dotnet test command generates those trx-files (it's not the first time). If it was working in the later versions I think we could just tell people to upgrade but since that is not the case atm I'm not sure what the best way forward is.

[GeorchW]

Do you know whether there is a TRX spec somewhere? I wonder what the difference between the properties UnitTest/@name and UnitTest/TestMethod/@name is. Maybe we should really use the first one.

I'll try to have a look at the code of the TRX writer in the .NET SDK, but I generally think that we need to file an issue against the SDK.

[GeorchW]

I haven't found a specification and the code is a little hard to navigate. I've opened an issue there.

[stefanforsberg]

By the looks of their response we might do well to start looking at the UnitTest/@name field.

We might be able to use functionality to split a FQN into parts then right? I'm not sure at the top of my head what the implications for our implementation would be.

And as you are alluding to in #269 it's hard to know if this field will be stable going forward.

[GeorchW]

We might be able to use functionality to split a FQN into parts then right?

We already do, partially.