CoreFX tests sometimes time out during PR validation builds

[dagood]

• https://dev.azure.com/dnceng/public/_build/results?buildId=522079&view=logs&j=935c00b8-5de8-597d-a783-42e7f079746c&t=a2eb024e-cab8-5a87-0588-93d2706bcaf2&l=266
• https://dev.azure.com/dnceng/public/_build/results?buildId=522713&view=logs&j=935c00b8-5de8-597d-a783-42e7f079746c&t=a2eb024e-cab8-5a87-0588-93d2706bcaf2&l=267

This seems to be flaky. /cc @crummel @adaggarwal

[adaggarwal]

I have also observed the same in CI. However, I don't see that whenever I try it locally.

[adaggarwal]

I saw this in one of the timeout cases

Resolving SDK 'Microsoft.NET.Sdk.IL'...

[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk.IL. Detected config(s) in env:
'ARCADE_CONFIGURATION' for 'Microsoft.DotNet.Build.Tasks.Configuration/1.0.0-beta.20113.5' at '/src/Tools/source-built/Microsoft.DotNet.Build.Tasks.Configuration/'
'IL' for 'Microsoft.NET.Sdk.IL/1.0.0-source-built' at '/src/Tools/source-built/Microsoft.NET.Sdk.IL/'
'ARCADE_COREFX_TESTING' for 'Microsoft.DotNet.CoreFxTesting/1.0.0-beta.20113.5' at '/src/Tools/source-built/Microsoft.DotNet.CoreFxTesting/'
'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/1.0.0-beta.20113.5' at '/src/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/'

[SourceBuiltSdkResolver] Overriding Microsoft.NET.Sdk.IL with 'IL'

Error when reading the file: System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.__Error.EndOfFile()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.Build.Logging.StructuredLogger.BuildEventArgsReader.ReadBuildEventArgsFields() in C:\MSBuildStructuredLog\src\StructuredLogger\BinaryLogger\BuildEventArgsReader.cs:line 628
   at Microsoft.Build.Logging.StructuredLogger.BuildEventArgsReader.ReadBuildMessageEventArgs() in C:\MSBuildStructuredLog\src\StructuredLogger\BinaryLogger\BuildEventArgsReader.cs:line 463
   at Microsoft.Build.Logging.StructuredLogger.BuildEventArgsReader.Read() in C:\MSBuildStructuredLog\src\StructuredLogger\BinaryLogger\BuildEventArgsReader.cs:line 95
   at Microsoft.Build.Logging.StructuredLogger.BinLogReader.Replay(Stream stream) in C:\MSBuildStructuredLog\src\StructuredLogger\BinaryLogger\BinLogReader.cs:line 78```

[dagood]

I believe that error is caused by an incomplete binlog, it's not indicating any build problem. So from that info it would seem the CI hangs (or dies?) after our SDK resolver replaces the IL SDK.

[adaggarwal]

I believe that error is caused by an incomplete binlog, it's not indicating any build problem. So from that info it would seem the CI hangs (or dies?) after our SDK resolver replaces the IL SDK.

Yep. And each timeout occurs at this point, which is interesting.

[dseefeld]

That's the point where it runs the corefx tests, I think. On a successful build, after that line, it runs for about 4 minutes:

2020-02-27T18:22:22.0647587Z       SOURCE_BUILT_SDK_DIR_IL=/src/Tools/source-built/Microsoft.NET.Sdk.IL/
2020-02-27T18:26:52.3198683Z   [18:26:52.31] Building 'corefx'...done

On a failing build, it takes about 2 hours and then times out:

2020-02-16T06:48:22.1658504Z       SOURCE_BUILT_SDK_DIR_IL=/src/Tools/source-built/Microsoft.NET.Sdk.IL/
2020-02-16T08:41:48.2414019Z ##[error]The task has timed out.

[MichaelSimons]

[Triage] CoreFX tests are no longer integrated into source-build. The unified build work for .NET 8.0 and 9.0 will make it possible to run any repo's unit tests against source-build.