MSTest 2.2.4 does not work with [DataTestMethod] [DynamicData] and a complex object

[TheFireCookie]

Description

Everything was working fine in 2.2.3. Then we updated to 2.2.4 and it does not work anymore.
So we have some classes with [DataTestMethod] and some dynamic data that we feed into it.
The tests were green in 2.2.3 and upgrading to 2.2.4 break them with this error:

C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug>dotnet test MsTest224DataTestBug.csproj
 Determining projects to restore...
 Restored C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\MsTest224DataTestBug.csproj (in 529 ms).
 MsTest224DataTestBug -> C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\bin\Debug\net5.0\MsTest224DataTestBug.dll
Test run for C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\bin\Debug\net5.0\MsTest224DataTestBug.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.10.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
An exception occurred while invoking executor 'executor://mstestadapter/v2': Type 'MsTest224DataTestBug.MyStringContainer' with data contract name 'MyStringContainer:http://schemas.datacontract.org/2004/07/MsTest224DataTestBug' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.
Stack trace:
  at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeAndVerifyType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, Boolean verifyKnownType, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
  at System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson.SerializeWithXsiTypeAtTopLevel(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle originalDeclaredTypeHandle, Type graphType)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObject(XmlWriterDelegator writer, Object graph)
  at System.Runtime.Serialization.XmlObjectSerializer.InternalWriteObject(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
  at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(XmlDictionaryWriter writer, Object graph)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(Stream stream, Object graph)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.DataSerializationHelper.Serialize(Object[] data)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.ToTestCase()
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.SendTestCases(String source, IEnumerable`1 testElements, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext, IMessageLogger logger)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.DiscoverTestsInSource(String source, IMessageLogger logger, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken cancellationToken)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
  at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
  at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)


Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

Steps to reproduce

Here is a full sample reproducing the bug:
https://github.com/TheFireCookie/MsTest224DataTestBug
In this repo, there is a folder MsTest223DataTestBug in which the command 'dotnet test' works correctly, the other folder is MsTest224DataTestBug and in this one, it's exactly the same code except I've upgraded MSTest to 2.2.4 and there a 'dotnet test' fails with the error written earlier.

Expected behavior

No regression and we can still use [DynamicData] with complex objects

Actual behavior

We cannot use anymore [DynamicData] with complex objects, only simple types like string, int etc

Environment

Tested and reproduced on Windows 10 and CentOS 7.9.
It's running on .NET 5.0.

AB#1334620

[Haplois]

Thank you for the report. This bug is caused because of a change in how we handle attributes derived from ITestDataSource.

I am working on a fix, and will release a preview Monday for this bug.

[sk6868]

It would be nice if regression tests for these cases were added in test/UnitTests

[Haplois]

@sk6868 we actually have tests, but apparently they are not complex enough.

[sk6868]

@Haplois yes I meant adding complex type (currently there only exists DynamicData tests using string and int types) use case tests to the existing regression tests :)

[Haplois]

This is fixed, and a preview is available in our daily feed. If you are not sure about how to connect to our daily feed please check out this page.

Fixed packages:

• MSTest.TestAdapter v2.2.5-preview-20210531-02
• MSTest.TestFramework v2.2.5-preview-20210531-02

Could you please test it and verify this is fixed the issue?

/cc: @TheFireCookie, @sk6868, @DOliana, @davesmits, @rikrak, @andessi, @Akinzekeel, @chriswelles

[Akinzekeel]

@Haplois thanks for the quick update.

I just tested the preview and re-ran all of my tests. The ones that were bugged out before (using types such as double and char) are working now. However a new bug appears to have popped up.

The following kind of tests are failing for me now (they worked previously in 2.2.3, not sure about 2.2.4):

[DataTestMethod]
[DataRow("john.doe@example.com", "abc123", null)]
[DataRow("john.doe@example.com", "abc123", "/unit/test")]
public async Task Can_SignIn(string Email, string Password, string ReturnUrl) { ... }

This is the error reported in the test explorer for these tests:

  Message: 
    Value cannot be null. (Parameter 'typeName')

  Stack trace: 
    RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
    Type.GetType(String typeName)

Update: there is now also an issue with byte-based enums (int-based enums work)

Message: 
    Test method My_Test_Method threw exception: 
    System.ArgumentException: Object of type 'System.Int32' cannot be converted to type 'MyByteBasedEnum'.

  Stack trace: 
    RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
    RuntimeType.CheckValue(Object value, Binder binder, CultureInfo culture, BindingFlags invokeAttr)
    MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
    RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    MethodBase.Invoke(Object obj, Object[] parameters)
    ThreadOperations.ExecuteWithAbortSafety(Action action)

[Haplois]

This is hopefully fixed, and a preview is available in our NuGet feed.

Fixed packages:

• MSTest.TestAdapter v2.2.5-preview-20210604-01
• MSTest.TestFramework v2.2.5-preview-20210604-01

Could you please test it and verify if this fixed the issue?

/cc: @TheFireCookie, @sk6868, @DOliana, @davesmits, @rikrak, @andessi, @Akinzekeel, @chriswelles, @conficient, @xantiva

[TheFireCookie]

So my test with the complex objects are working now.

I've also tested the method of @Akinzekeel and it seems to work fine too.

When can we expect a real release of MSTest with the fixed issues?
Have a nice day!

[ybru-isiware]

Hi,

I try the v2.2.5-preview-20210604-01, but my problem persists.

I create a sample here : https://github.com/ybru-isiware/MsTest224DataTestBug.

This sample work well in 2.2.3.

Sample content

In the sample, i create a User class and a UserService class.
The UserService have one methode named ParseUserDatas who take a string for extract user information.

In my test project, i create a DataTestMethod and use DynamicDataAttribute.

The test method take two parameters :

[DataTestMethod()]
[DynamicData(nameof(GetParseUserDatas), DynamicDataSourceType.Method)]
public void ParseUserDatasTest(string userDatas, User expectedUser)
{
   // ...
}

The datas look like that :

public static IEnumerable<object[]> GetParseUserDatas()
{
    yield return new object[] {
        "John;Doe",
        new User()
        {
            FirstName = "John",
            LastName = "Doe"
        }
    };
}

Error :

 ParseUserDatasTest (John;Doe,MsTest224DataTestBugLib.User)
   Source: UserServiceTests.cs line 17
   Duration: < 1 ms

  Message: 
    Element ':root' contains data of the 'http://schemas.datacontract.org/2004/07/MsTest224DataTestBugLib:User' data contract. The deserializer has no knowledge of any type that maps to this contract. Add the type corresponding to 'User' to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding it to the list of known types passed to DataContractSerializer.
  Stack Trace: 
    XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
    XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)
    DataContractJsonSerializerImpl.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName)
    XmlObjectSerializer.InternalReadObject(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
    XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
    DataContractJsonSerializerImpl.ReadObject(XmlDictionaryReader reader)
    DataContractJsonSerializerImpl.ReadObject(Stream stream)

Have a nice day !

[Akinzekeel]

v2.2.5-preview-20210604-01 fixed almost all cases for me, but byte-enums are still failing.

Test method signature:

[DataTestMethod]
[DoNotParallelize]
[DataRow(ShipmentThresholdPolicy.AdvancePaymentRequired, false, false)]
[DataRow(ShipmentThresholdPolicy.PackBeforePayment, true, false)]
[DataRow(ShipmentThresholdPolicy.PackAndShipBeforePayment, true, true)]
public async Task Publish_Sets_PackBefore_Properties(ShipmentThresholdPolicy policy, bool canPackBeforePayment, bool canShipBeforePayment) { ... }

Enum definition:

public enum ShipmentThresholdPolicy : byte
{
    AdvancePaymentRequired,
    PackBeforePayment,
    PackAndShipBeforePayment
}

Error:
System.ArgumentException: Object of type 'System.Int32' cannot be converted to type '(redactedNamespace).ShipmentThresholdPolicy'.

[PRRB]

When testing update to 2.2.4 I ran into an issue with string[] and int[] params in DataRow attributes causing test-discovery exception. Developer Community ticket someone posted here:
https://developercommunity.visualstudio.com/t/cant-set-arrayofstring-as-the-datarow-in-mstest/1435155

Also I had an issue with floating point value params (1.5, 1.5f, 1.5d) being casted to decimal value and causing test failures.

I've run our tests against v2.2.5-preview-20210604-01 posted above - both of the issues I ran into in 2.2.4 seem to be resolved.

[Haplois]

@Akinzekeel
v2.2.5-preview-20210604-01 fixed almost all cases for me, but byte-enums are still failing.

I tried to reproduce this, but my tests are passing. Could you clean your solution, make sure you're using the latest preview and try again?

[Haplois]

Thank you @ybru-isiware,

I was able to repro the issue. Working on it.

[Akinzekeel]

@Haplois I found out that the test works when I use the command line (dotnet test), however it keeps failing in Visual Studio no matter what. I've tried cleaning & rebuilding and also deleted the .vs folder, restarted Visual Studio, cleared the test results cache but the test still fails.

I'm using these versions for the test project:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.2.5-preview-20210604-02" />
<PackageReference Include="MSTest.TestFramework" Version="2.2.5-preview-20210604-02" />

Do you have any idea what could be wrong?

[Haplois]

@Akinzekeel, could you try cleaning and rebuilding the solution? You also might need to restart VS.

[Haplois]

With #864, last failing case is fixed too. We also added some tests. @Akinzekeel, can you verify?

Fixed packages:

• MSTest.TestAdapter v2.2.5-preview-20210605-01
• MSTest.TestFramework v2.2.5-preview-20210605-01

[Akinzekeel]

With #864, last failing case is fixed too. We also added some tests. @Akinzekeel, can you verify?

Fixed packages:

• MSTest.TestAdapter v2.2.5-preview-20210605-01
• MSTest.TestFramework v2.2.5-preview-20210605-01

@Haplois I can confirm that this version fixed all of my problems. Thank you!

[Haplois]

Thank you for confirming @Akinzekeel, closing the issue.

[japj]

@Haplois

With #864, last failing case is fixed too. We also added some tests.

Is there a timeline on when 2.2.5 is planned to be released?