[Regression] Tests failed: System.IO.Compression.Brotli.Performance.Tests - 20min timeout on Windows

[karelz]

System.IO.Compression.Brotli.Performance.Tests test suite started failing with 'fire' on Windows platforms in build 20180120.01.
Looking at a few logs it seems that xunit doesn't print anything after initial output and infra kills it after 20 min:

Discovering: System.IO.Compression.Brotli.Performance.Tests
Discovered:  System.IO.Compression.Brotli.Performance.Tests
Starting:    System.IO.Compression.Brotli.Performance.Tests

Note Ubuntu17.04 'fire' failures are across all tests (the image went EOL), please ignore it. The platform should be removed from our test run matrix on 1/25.

History of failures

Day	Build	Failures
1/20	20180120.01	4x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x86)
1/20	20180120.02	4x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x86)
1/21	20180121.01	4x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x86)
1/21	20180121.02	3x (Win10-x86 & Win10.Core-x86 & Win7-x86)
1/22	20180122.01	4x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x86)
1/22	20180122.02	4x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x64)
1/23	20180123.01	3x (Win10.Core-x86 & Win7-x86 & Win8.1-x64)
1/23	20180123.04	4x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x64)
1/24	20180124.02	5x (Win10-x86 & Win10.Core-x86 & Win7-x64 & Win7-x86 & Win8.1-x86)
1/24	20180124.06	1x (Win8.1-x64) - note: -x86 runs didn't execute in this build
1/25	20180125.01	3x (Win10-x86 & Win10.Core-x86 & Win7-x86)
1/25	20180125.06	5x (Win10-x86 & Win10.Core-x86 & Win7-x64 & Win7-x86 & Win8.1-x86)
1/25	20180125.08	5x (Win10-x86 & Win10.Core-x86 & Win7-x64 & Win7-x86 & Win8.1-x86)
1/26	20180126.03	5x (Win10-x86 & Win10.Core-x86 & Win7-x64 & Win7-x86 & Win8.1-x86)
1/27	20180127.01	5x (Win10-x86 & Win10.Core-x86 & Win7-x64 & Win7-x86 & Win8.1-x86)
1/27	20180127.04	5x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x64 & Win8.1-x86)
1/28	20180128.01	5x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x64 & Win8.1-x86)
1/28	20180128.04	5x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x64 & Win8.1-x86)
1/29	20180129.01	5x (Win10-x86 & Win10.Core-x86 & Win7-x64 & Win7-x86 & Win8.1-x86)
1/29	20180129.05	1x (Win8.1-x64) - note: -x86 runs didn't execute in this build
1/30	20180130.01	5x (Win10-x86 & Win10.Core-x86 & Win7-x64 & Win7-x86 & Win8.1-x86)
1/30	20180130.05	5x (Win10-x86 & Win10.Core-x86 & Win7-x86 & Win8.1-x64 & Win8.1-x86)

[karelz]

@ianhays can you please take a look?

[ianhays]

Hmm. It's possible I added too many tests and they're just hitting the timeout limit. They do take a very long time on my machine. I'll see exactly how long, and shorten that if necessary to get them completing faster.

[karelz]

That might be the reason - successful run of Win8.1-x86 in 20180121.02 claims 14:02 time (looking at timestamps it is 16:30) - quite close to the 20 min timeout.

[ianhays]

Yup, that's the reason. I've been running the perf tests for like an hour now and they're still chugging along.

[danmoseley]

@Anipik I guess this is not true (no dumps uploaded -- no URL shown) and you are fixing?

2018-01-30 02:42:34,424: INFO: servicebusrepository(97): renew_workitem_lock: renew_workitem_lock Status Code: 200
2018-01-30 02:43:05,845: INFO: job(70): _create_and_upload_dump: Creating Dumps Process Exit Code: 0
2018-01-30 02:43:09,595: INFO: job(72): _create_and_upload_dump: Uploading Dumps Process Exit Code: 1
2018-01-30 02:43:09,595: INFO: job(73): _create_and_upload_dump: Dumps Created and Uploaded Successfully
2018-01-30 02:43:09,595: ERROR: job(43): kill: Job running for too long. Killing...

[danmoseley]

@ianhays do you expect to have a fix soon? It hung in several configurations in both builds today.

https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20180130.01/workItem/System.IO.Compression.Brotli.Performance.Tests/wilogs

[Anipik]

@danmosemsft the dumps are not being uploaded because the upload website is under construction as we are adding an authentication layer on the website.

[Anipik]

The pr is up for showing the URL and I am waiting for matt`s approval for that

[karelz]

@ianhays alternatively, you can disable the tests, until the underlying problem, is addressed

[Priya91]

failed in PR run dotnet/corefx#26666

[stephentoub]

This is taking out a bunch of outerloop runs. I'm going to disable them.