recent CI failure

[frank-dong-ms-zz]

we are start seeing some ci failure recently:

https://dev.azure.com/dnceng/public/_build/results?buildId=709641&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=706523&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=706453&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=699015&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=689649&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=656357&view=results

there are several issues here:

1. error CS7035: The specified version string does not conform to the recommended format - major.minor.build.revision

https://dev.azure.com/dnceng/public/_build/results?buildId=656357&view=results

this also happens on local randomly

2. fatal: unable to access 'https://github.com/mongodb/homebrew-brew/': Failed to connect to github.com port 443: Operation timed out

Error: Fetching /usr/local/Homebrew/Library/Taps/mongodb/homebrew-brew failed!
https://dev.azure.com/dnceng/public/_build/results?buildId=709641&view=logs&j=4b233af4-7b14-5f68-27c6-9c4d7ac87519&t=6c4d0d4f-93f4-59fe-bb3a-eecc956920fe

looks like random failure related to github authentication

3. SavePipePValue failure, new random failure

https://dev.azure.com/dnceng/public/_build/results?buildId=706523&view=logs&j=41509eb4-74ce-5e57-61b4-bdf74b39e7c1&t=522d178a-829f-5bff-ccb9-04bea054b64d

4. benchmark failure, new random failure

https://dev.azure.com/dnceng/public/_build/results?buildId=706453&view=logs&j=4b233af4-7b14-5f68-27c6-9c4d7ac87519&t=c45ec5f3-1036-5f06-ba3b-dc6ad6175c22

[xUnit.net 00:02:08.63] Microsoft.ML.Benchmarks.Tests.BenchmarksTest.BenchmarksProjectIsNotBroken(type: typeof(Microsoft.ML.Benchmarks.StochasticDualCoordinateAscentClassifierBench)) [FAIL]
// ***** BenchmarkRunner: Start *****
// ***** Found 2 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start *****
X Microsoft.ML.Benchmarks.Tests.BenchmarksTest.BenchmarksProjectIsNotBroken(type: typeof(Microsoft.ML.Benchmarks.StochasticDualCoordinateAscentClassifierBench)) [1m 8s]
Error Message:
All reports should have at least one "ExecuteResult" with "FoundExecutable" = true and at least one "Data" item
Expected: True
Actual: False
Stack Trace:
at Microsoft.ML.Benchmarks.Tests.BenchmarksTest.BenchmarksProjectIsNotBroken(Type type) in /Users/runner/runners/2.170.1/work/1/s/test/Microsoft.ML.Benchmarks.Tests/BenchmarksTest.cs:line 69

5. some timeouts failure

https://dev.azure.com/dnceng/public/_build/results?buildId=689649&view=logs&j=dd8eddb6-ecc6-5f65-73e6-df90e5693b94
https://dev.azure.com/dnceng/public/_build/results?buildId=706453&view=logs&j=87172896-2df6-55a2-04c3-60b48f00f19f

6. TestCancellation hanging again

https://dev.azure.com/dnceng/public/_build/results?buildId=699015&view=logs&j=9d6f93fc-7103-540d-abb9-b79bad552b0d&t=febb8f43-1f98-59b4-5544-6955325789bd

7. Microsoft.Extensions.ML.FileLoaderTests.can_reload_model: FileLoader ChangeToken didn't fire before the allotted time.

https://dev.azure.com/dnceng/public/_build/results?buildId=716174&view=logs&j=41509eb4-74ce-5e57-61b4-bdf74b39e7c1&t=522d178a-829f-5bff-ccb9-04bea054b64d

[frank-dong-ms-zz]

address issue 5: #5265

[frank-dong-ms-zz]

issue 1 looks like a bug in msbuild versioning. This only happens when the build is crossing 2 days as msbuild versioning is generating versioning file each day so when some project starts to build when the system time is crossing to new day, seems msbuild versioning can't handle this case and read empty BuildNumberMajor and BuildNumberMinor which cause the assembly version number to be a invalid format.

I can repro this issue by set system time to a time very close to midnight, say 11:58 PM then start build from cmd, there is pretty good chance to repro the build version issue. Since we don't have a way to upgrade build tools now I will use default value of BuildNumberMajor and BuildNumberMinor to mitigate this issue.

a mitigation for issue 1: #5268

[frank-dong-ms-zz]

issue 2 looks like random authentication service lag and is not in our control.

[frank-dong-ms-zz]

currently no repro on issue 3, 4 and 6

[frank-dong-ms-zz]

issue 7 fixed with longer wait time

[frank-dong-ms-zz]

checked CI history of past 2 month and I'm not seeing any of similar issues before so close this issue for now. Will reopen and investigate if we see more similar failures.