[VsTest task] Vstest.console.exe error "An existing connection was forcibly closed by the remote host"

[TestO2015]

Environment

• Server - Azure Pipelines
• Agent - Hosted

Issue Description

We have a Visual Studio Test task which fails with an error returned from vstest.console.exe.
There were no test failures reported however not all were run; it appears it aborted before they completed.

Error from VsTest log:

2018-12-13T00:33:18.5763881Z Total tests: Unknown. Passed: 146. Failed: 0. Skipped: 0.
2018-12-13T00:33:18.5763946Z Test Run Aborted.
2018-12-13T00:33:18.5763992Z Test execution time: 1.7709 Minutes
2018-12-13T00:33:18.5765323Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2018-12-13T00:33:18.6217988Z ##[error]Error: C:\hostedtoolcache\windows\VsTest\15.9.0\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe failed with return code: 1
2018-12-13T00:33:19.1708369Z ##[error]VsTest task failed.

Error from vstest.console.exe:

2018-12-12T07:09:25.4749058Z TpTrace Error: 0 : 720, 12, 2018/12/12, 07:08:55.736, 15811144290, vstest.console.exe, Socket: Message loop: failed to receive message due to socket error System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
2018-12-12T07:09:25.4749141Z at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
2018-12-12T07:09:25.4749196Z at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

2018-12-12T07:09:25.4749264Z --- End of inner exception stack trace ---
2018-12-12T07:09:25.4749312Z at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
2018-12-12T07:09:25.4749433Z at System.IO.Stream.ReadByte()
2018-12-12T07:09:25.4749495Z at System.IO.BinaryReader.ReadByte()
2018-12-12T07:09:25.4749548Z at System.IO.BinaryReader.Read7BitEncodedInt()
2018-12-12T07:09:25.4749610Z at System.IO.BinaryReader.ReadString()
2018-12-12T07:09:25.4749663Z at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
2018-12-12T07:09:25.4749739Z at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken

VisualStudioTestTask_log_18_5820.zip
TestRunLog_log_19_5820.zip

[ShreyasRmsft]

@TestO2015 does this repro constantly with every run?

Does this happen if you run the tests on a self hosted agent also?

Can you provide a sample repro project if possible?

[TestO2015]

@ShreyasRmsft
The issue is intermittent occurring in ~30% of the builds. We haven't tried it on a self hosted agent.

Please let me know if there's more info we can provide or if you have any ideas how we could diagnose it ourselves.

Thanks.

[vagisha-nidhi]

@TestO2015
This seems like the test host is crashing due to some reason. There is an advanced diagnostics feature which collects dump during a crash. This is pre-selected by default. If there are aborted runs because of test crashes, you will see the test run message and attachments, pointing to the faulty test cases. You can turn on collect dump as "Always" and share the memory dump.

Can you try this with the latest version of test platform? You can use the platform installer task to do this.

[vagisha-nidhi]

@TestO2015 Any updates?

[kaadhina]

Closing this since we do not have enough info to investigate this further. Please reopen once you have the data asked above if you are still facing this.

[TestO2015]

Apologies for the delay, this issue is still occurring, could we please reopen this?

Here's what we've tried:

Enabled 'Collect advanced diagnostics in case of test failure'
> No test failures were reported, but not all were run.

Set 'Collect Process Dump and attach to report to' 'Always'
> Where does this get attached to?

Ran the vstest.console.exe process with the /diag option and printed the results (already attached above in TestRunLog_log_19_5820.zip)
> vstest.console.exe, Socket: Message loop: failed to receive message due to socket error System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

Used the latest version of the test platform, which didn't solve the issue.

Thank you

[prawalagarwal]

@TestO2015 Can you try out this:

1. Use the distributed mode of execution. You can do this in the advanced settings of the VsTest task
   
   You can specify any batch size. Dump collection works only in distributed mode. Apologies to have missed that in the earlier reply.
2. Make sure you are on 15.9 platform and above for the above step to work. You can use the platform installer task to ensure this, by selecting "latest stable"
   
   But make sure you have selected Test Platform version as installed by tools installer in the vstest task:
   

Now, once you are done with this, if a crash happens again, the dumps are attached as Test Run attachments (click on the test run in test summary tab):

[prawalagarwal]

@TestO2015 any updates here?

[TestO2015]

Thank you @prawalagarwal .

After enabling a custom batch size, test failures were reported with an error during the MSTEST cleanup routine of our tests. We've fixed the error during cleanup and it now works.

It appears that the SocketException error message was hiding actual test failures. If so that's something to fix.

[prawalagarwal]

Thanks for reporting back!