vstest.console.exe fails test run with socket exception

[NateB2]

Description

Seemingly at random, our test run will fail with a socket connection error as follows:

vstest.console.exe, ProxyDataCollectionManager.InvokeDataCollectionServiceAction: TestPlatformException = System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.DataCollection.DataCollectionRequestSender.SendAfterTestRunStartAndGetResult(ITestMessageEventHandler runEventsHandler, Boolean isCancelled)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyDataCollectionManager.<>c__DisplayClass15_0.<AfterTestRunEnd>b__0()
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyDataCollectionManager.InvokeDataCollectionServiceAction(Action action, ITestMessageEventHandler runEventsHandler).

We never received this error until we migrated to 15.6.

Steps to reproduce

I'm not sure how to reproduce this reliably. We run it on an Azure VM with multiple VSTS agent instances installed, and it seems to happen more frequently when the VM resources are taxed, but occasionally it occurs when no other instances are performing tasks.

Expected behavior

Test run finishes successfully

Actual behavior

Test run fails

Diagnostic logs

Here are the diagnostic logs surrounding the error. If there is any other diagnostic information you need, let me know. https://gist.github.com/NateB2/f97f8da22492cfb4489ddef87a0946e2

Environment

Windows Server 2016, VS 2017 15.6, VSTS agent 2.129.1.

[ravarnamsft]

We are seeing a similar issue on VSTS. This fails the test run though all tests pass. Can we disable this datacollection via a parameter in runsettings or commandline argument to vstest

[stefanmsft]

Also seeing this IOException on VSTS, have been encountering it for the last two days. It seems to happen pretty frequently on builds - occasionally the exception is not thrown and the build succeeds. We're seeing the exact same exception as above.

[Aurelien-Thomas]

I currently got it 100% of the time on VSTS, sometimes 2 or 3 times per build when running unit tests

[Diminouille]

same thing here, we got it systematically. Our build pipeline is currently broken, goodbye continous feedback :(

[smadala]

Primary investigation indicates this issue occurs when running datacollection(Code Coverage, Fakes, etc..) in parallel. To unblock, Please disable the parallel(MaxCpuCount in runsettings) option while running tests. I'm working on fix. Will let you know once fix is available.

[smadala]

Published Microsoft.TestPlatform.15.6.2 package with fix. You can get the latest test platform by using the Visual studio test platform installer task to try out the fix right away.

We are planning to fix this issue in VS 15.6 upcoming update.

[NateB2]

We have ran builds with the fixed version since Monday, and no builds have failed due to the socket exception, so it appears to be fixed for us. Thanks!

[TestO2015]

We're experiencing the same socket exception intermittently, in ~30% of builds. Vstest.console.exe_SocketException.txt

Environment details: We're running VsTest in an Azure Pipeline on a Hosted Agent. We've tried VsTest versions 16.0.0, 15.8.0.

We have an issue raised as well with more details in: microsoft/azure-pipelines-tasks#9095

[sroe]

Same here, using VS 2017 Test Platform 15.9.1.2019030503, I have to kill the vstest.console.exe-Processes to get another Unittest running.
Sometimes there are 2 processes spawned but I only run 1 Method with Test Explorer.
I will update von VS 2019 in a week and will look if it's buggy there too.
As a workaround I am adding PostBuild-Events to the Unit Test Project with cmd.exe /K "kill vstest.console.exe" but thats realy a hack, because its not with every testrun, sometimes you run 200 tests without a problem...

[TestO2015]

Hi @sroe,

We were able to resolve this issue in microsoft/azure-pipelines-tasks#9095.

The underlying error was covered up by the vstest.console.exe SocketException. And perhaps strangely, the underlying exception could be revealed by enabling a custom batch size. The underlying error was on our end.