[Windows] Test failures: The operation has been cancelled / The handle is invalid / etc.

[karelz]

Types of failures

Affected tests:

Test	#	Test type / notes
MaxResponseContentBufferSize_ThrowsIfTooSmallForContent	11x	System.Net.Http.Functional.Tests.HttpClientTest / System.Net.Http.Functional.Tests.HttpProtocolTests
GetAsync_SupportedSSLVersion_Succeeds	9x	System.Net.Http.Functional.Tests.HttpClientHandler_SslProtocols_Test Note: also known dependency on SslLabs remote server - #24433)
GetAsync_RedirectResponseHasCookie_CookieSentToFinalUri	7x	System.Net.Http.Functional.Tests.HttpClientHandlerTest
PostAsync_EmptyContent_ContentTypeHeaderNotSent	6x	System.Net.Http.Functional.Tests.PostScenarioTest
GetAsync_SetCookieContainer_CookieSent	6x	System.Net.Http.Functional.Tests.PlatformHandler_HttpCookieProtocolTests
GetAsync_TrailingHeaders_Ignored	5x	System.Net.Http.Functional.Tests.HttpClientHandlerTest
PostRewindableContentUsingAuth_NoPreAuthenticate_Success	5x	System.Net.Http.Functional.Tests.PostScenarioTest
ThresholdExceeded_ThrowsException	5x	System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test
PostAsync_CallMethod_EmptyContent	4x	System.Net.Http.Functional.Tests.HttpClientHandlerTest
PostUsingChunkedEncoding_Success	4x	System.Net.Http.Functional.Tests.PostScenarioTest
System.Net.Http.Functional.Tests.PostScenarioTest
PostLongerContentLengths_UsesChunkedSemantics	4x	System.Net.Http.Functional.Tests.PostScenarioTest
PostSyncBlockingContentUsingChunkedEncoding_Success	3x	System.Net.Http.Functional.Tests.PostScenarioTest
GetStreamAsync_ReadZeroBytes_Success	3x	System.Net.Http.Functional.Tests.ResponseStreamTest
NoCallback_ValidCertificate_CallbackNotCalled	3x	System.Net.Http.WinHttpHandlerFunctional.Tests.ServerCertificateTest
PostEmptyContentUsingConflictingSemantics_Success	3x	System.Net.Http.Functional.Tests.PostScenarioTest
GetAsync_SetCookieContainer_CookieSent	3x	System.Net.Http.Functional.Tests.HttpCookieProtocolTests
PostUsingNoSpecifiedSemantics_UsesChunkedSemantics	2x	System.Net.Http.Functional.Tests.PostScenarioTest
GetAsync_Max1_ConcurrentCallsStillSucceed	1x	System.Net.Http.Functional.Tests.HttpClientHandler_MaxConnectionsPerServer_Test
Timeout_SetTo30AndGetResponseFromLoopbackQuickly_Success	1x	System.Net.Http.Functional.Tests.HttpClientTest
GetAsync_UseResponseHeadersReadAndCallLoadIntoBuffer_Success	1x	System.Net.Http.Functional.Tests.ResponseStreamTest
PostEmptyContentUsingChunkedEncoding_Success	1x	System.Net.Http.Functional.Tests.PostScenarioTest
PostRepeatedFlushContentUsingChunkedEncoding_Success	1x	System.Net.Http.Functional.Tests.PostScenarioTest
PostAsync_CallMethod_NullContent	1x	System.Net.Http.Functional.Tests.HttpClientHandlerTest
GetAsync_UnknownRequestVersion_ThrowsOrDegradesTo11	1x	System.Net.Http.Functional.Tests.HttpProtocolTests_Dribble
HttpClientHandler_Authentication_Succeeds	1x	System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test
HttpClientHandler_IncorrectCredentials_Fails	1x	System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test
SendAsync_SendSameRequestMultipleTimesDirectlyOnHandler_Success	1x	System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandlerTest

Similar to bugs:

• Test failure: System.Net.Http.Functional.Tests.HttpClientHandler_MaxConnectionsPerServer_Test / GetAsync_MaxLimited_ConcurrentCallsStillSucceed #24446 - GetAsync_MaxLimited_ConcurrentCallsStillSucceed
• Test failure: System.Net.Http.Functional.Tests.HttpClientHandlerTest / PostAsync_CallMethod_StreamContent #24449 - PostAsync_CallMethod_StreamContent
• Test failure: System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test / UseCallback_NotSecureConnection_CallbackNotCalled #24454 - UseCallback_NotSecureConnection_CallbackNotCalled
• Test failure: System.Net.Http.Functional.Tests.HttpProtocolTests / GetAsync_ExpectedStatusCodeAndReason_Success #24434 - GetAsync_ExpectedStatusCodeAndReason_Success
• Test failure: System.Net.Http.Functional.Tests.ResponseStreamTest / GetStreamAsync_ReadToEnd_Success #24486 - GetStreamAsync_ReadToEnd_Success
• Test failure: System.Net.Http.Functional.Tests.HttpClientHandler_SslProtocols_Test / GetAsync_AllowedSSLVersion_Succeeds #24485 - GetAsync_AllowedSSLVersion_Succeeds
• Test failure: System.Net.Http.Functional.Tests.HttpClientHandlerTest / PostAsync_ReuseRequestContent_Success #24450 - PostAsync_ReuseRequestContent_Success
• Test failure: System.Net.Http.Functional.Tests.HttpClientHandlerTest / GetAsync_ServerNeedsNonStandardAuthAndSetCredential_StatusCodeUnauthorized #23661 - GetAsync_ServerNeedsNonStandardAuthAndSetCredential_StatusCodeUnauthorized
• Test failure: System.Net.Http.Functional.Tests.ResponseStreamTest / ReadAsStreamAsync_ValidServerResponse_Success #21838 - ReadAsStreamAsync_ValidServerResponse_Success
• Test failure: System.Net.Http.Functional.Tests.HttpClientHandler_DefaultProxyCredentials_Test / ProxyExplicitlyProvided_DefaultCredentials_Ignored #24487 - ProxyExplicitlyProvided_DefaultCredentials_Ignored
• Test failure: System.Net.Http.Functional.Tests.PostScenarioTest / PostUsingContentLengthSemantics_Success #24483 - PostUsingContentLengthSemantics_Success
• Test failure: System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test / SetDelegate_ConnectionSucceeds #23662 - SetDelegate_ConnectionSucceeds

The operation has been canceled

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.Net.Http.WinHttpException : The operation has been canceled

The handle is in the wrong state for the requested operation

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.Net.Http.WinHttpException : The handle is in the wrong state for the requested operation

The handle is invalid

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.Net.Http.WinHttpException : The handle is invalid

IOException: The handle is invalid

System.Net.Http.HttpRequestException : Error while copying content to a stream.
---- System.IO.IOException : The write operation failed, see inner exception.
-------- System.Net.Http.WinHttpException : The handle is invalid

The operation timed out

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.Net.Http.WinHttpException : The operation timed out

History of failures

Day	Build	OS	Error message	Test
5/9	20170509.01	Win7	The handle is in the wrong state for the requested operation	PostUsingChunkedEncoding_Success
5/9	20170509.01	Win7	The handle is invalid	Timeout_SetTo30AndGetResponseFromLoopbackQuickly_Success
5/15	20170515.02	Win10	The operation has been canceled	GetAsync_UseResponseHeadersReadAndCallLoadIntoBuffer_Success
5/16	20170516.02	Win10	The operation has been canceled	GetAsync_SupportedSSLVersion_Succeeds
5/21	20170521.02	Win10	The operation has been canceled	GetAsync_Max1_ConcurrentCallsStillSucceed
5/21	20170521.02	Win7	The operation has been canceled	GetAsync_TrailingHeaders_Ignored
5/26	20170526.01	Win7	The handle is invalid	PostLongerContentLengths_UsesChunkedSemantics
5/30	20170530.06	Win10	The operation has been canceled	PostUsingChunkedEncoding_Success
5/31	20170601.01	Win10	The operation has been canceled	PostAsync_CallMethod_EmptyContent
6/7	20170607.01	Win7	The handle is invalid	PostSyncBlockingContentUsingChunkedEncoding_Success
6/12	20170612.02	Win7	The operation has been canceled	PostAsync_EmptyContent_ContentTypeHeaderNotSent
6/16	20170616.01	Win7	The handle is invalid	PostAsync_CallMethod_EmptyContent
6/22	20170622.03	Win8.1	The operation has been canceled	GetStreamAsync_ReadZeroBytes_Success
10/3	20171003.01	Win7	The operation has been canceled	PostAsync_CallMethod_EmptyContent
10/8	20171008.02	Win7	The operation has been canceled	NoCallback_ValidCertificate_CallbackNotCalled
10/8	20171008.02	Win8.1	The operation has been canceled	NoCallback_ValidCertificate_CallbackNotCalled
10/15	20171015.02	Win7	The operation has been canceled	PostSyncBlockingContentUsingChunkedEncoding_Success
10/19	20171019.02	Win7	The operation has been canceled	GetAsync_SupportedSSLVersion_Succeeds
10/20	20171020.01	Win8.1	The operation has been canceled	GetAsync_RedirectResponseHasCookie_CookieSentToFinalUri
10/20	20171020.01	Win8.1	The operation has been canceled	PostAsync_EmptyContent_ContentTypeHeaderNotSent
10/22	20171022.01	Win7	The handle is invalid	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent
10/22	20171022.02	Win7	The handle is invalid	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent
10/23	20171023.01	Win10	The operation has been canceled	PostUsingNoSpecifiedSemantics_UsesChunkedSemantics
10/25	20171025.08	Win7	The handle is invalid	PostRewindableContentUsingAuth_NoPreAuthenticate_Success
10/30	20171030.01	Win7	The handle is invalid	PostRewindableContentUsingAuth_NoPreAuthenticate_Success
10/30	20171030.02	Win7	The handle is invalid	GetAsync_SupportedSSLVersion_Succeeds
11/3	20171103.07	Win7	The operation has been canceled	GetStreamAsync_ReadZeroBytes_Success
11/4	20171104.04	Win7	The operation has been canceled	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent
11/6	20171106.01	Win7	The handle is invalid	PostRewindableContentUsingAuth_NoPreAuthenticate_Success
11/7	20171107.02	Win7	The handle is in the wrong state for the requested operation	GetAsync_SupportedSSLVersion_Succeeds
11/8	20171108.01	Win7	The operation has been canceled	PostEmptyContentUsingConflictingSemantics_Success
11/15	20171115.01	Win7	The handle is in the wrong state for the requested operation	GetAsync_SupportedSSLVersion_Succeeds
11/15	20171115.01	Win7	The operation has been canceled	GetAsync_RedirectResponseHasCookie_CookieSentToFinalUri
11/15	20171115.02	Win7	The handle is invalid	PostAsync_EmptyContent_ContentTypeHeaderNotSent
11/15	20171115.03	Win7	The handle is invalid	GetAsync_TrailingHeaders_Ignored
11/16	20171116.54	Win7	The operation has been canceled	PostSyncBlockingContentUsingChunkedEncoding_Success
11/18	20171118.04	Win7	The handle is invalid	GetStreamAsync_ReadZeroBytes_Success
11/18	20171118.04	Win8.1	The operation has been canceled	GetAsync_TrailingHeaders_Ignored
11/20	20171120.01	Win10	The operation has been canceled	PostLongerContentLengths_UsesChunkedSemantics
11/20	20171120.01	Win7	The handle is invalid	GetAsync_TrailingHeaders_Ignored
11/22	20171122.01	Win10	The operation has been canceled	PostEmptyContentUsingChunkedEncoding_Success
11/22	20171122.01	Win8.1	The operation has been canceled	PostAsync_CallMethod_EmptyContent
11/28	20171128.07	Win10	The operation has been canceled	PostLongerContentLengths_UsesChunkedSemantics
11/29	20171129.02	Win8.1	The operation has been canceled	PostAsync_EmptyContent_ContentTypeHeaderNotSent
12/8	20171208.01	Win7	The handle is invalid	PostAsync_EmptyContent_ContentTypeHeaderNotSent
12/13	20171213.02	Win8.1	The operation has been canceled	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent
12/13	20171213.08	Win7	IOException: The handle is invalid	PostUsingChunkedEncoding_Success
12/15	20171215.05	Win7	The handle is in the wrong state for the requested operation	GetAsync_SupportedSSLVersion_Succeeds
12/16	20171216.01	Win7	The operation has been canceled	GetAsync_RedirectResponseHasCookie_CookieSentToFinalUri
12/16	20171216.03	Win8.1	The operation has been canceled	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent
12/17	20171217.01	Win7	The handle is invalid	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent
12/23	20171223.02	Win7	The handle is invalid	GetAsync_RedirectResponseHasCookie_CookieSentToFinalUri - only in catastrophic failure log
12/31	20171231.02	Win7	The handle is in the wrong state for the requested operation	GetAsync_SupportedSSLVersion_Succeeds - link
1/1	20180101.01	Win8.1	The operation has been canceled	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent - link
1/2	20180102.01	Win8.1	The operation has been canceled	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent - link
1/6	20180106.01	Win7	The operation has been canceled	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent - link
1/10	20180110.03	Win7	The operation has been canceled	NoCallback_ValidCertificate_CallbackNotCalled - link
1/14	20180114.01	Win7	The handle is invalid	PostAsync_EmptyContent_ContentTypeHeaderNotSent - link
1/14	20180114.03	Win7	The handle is in the wrong state for the requested operation	PostEmptyContentUsingConflictingSemantics_Success - link
1/16	20180116.05	Win7	The operation has been canceled	GetAsync_RedirectResponseHasCookie_CookieSentToFinalUri - link
1/18	20180118.04	Win7	The handle is invalid	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent - link
1/21	20180121.01	Win7	The handle is invalid	MaxResponseContentBufferSize_ThrowsIfTooSmallForContent - link
1/23	20180123.01	Win7	The handle is invalid	PostRewindableContentUsingAuth_NoPreAuthenticate_Success - link
1/23	20180123.01	Win7	The operation has been canceled	GetAsync_SupportedSSLVersion_Succeeds - link
1/27	20180127.01	Win7	The handle is invalid	GetAsync_TrailingHeaders_Ignored - link
1/27	20180127.04	Win10	The handle is invalid	PostRepeatedFlushContentUsingChunkedEncoding_Success - link
1/27	20180127.04	Win7	The handle is invalid	GetAsync_RedirectResponseHasCookie_CookieSentToFinalUri - only in catastrophic failure logs
1/28	20180128.04	Win10	The operation has been canceled	PostAsync_CallMethod_NullContent - link
1/30	20180130.05	Win7	The handle is invalid	PostEmptyContentUsingConflictingSemantics_Success - only in catastrophic failure logs
2/1	20180201.01	Win7	The handle is invalid	PostUsingChunkedEncoding_Success - link
2/1	20180201.01	Win7	The operation has been canceled	GetAsync_RedirectResponseHasCookie_CookieSentToFinalUri - link
2/9	20180209.05	Win7	The handle is invalid	ThresholdExceeded_ThrowsException - link
2/9	20180209.05	Win7	The handle is in the wrong state for the requested operation	GetAsync_UnknownRequestVersion_ThrowsOrDegradesTo11 - link
2/13	20180213.01	Win7	The handle is invalid	ThresholdExceeded_ThrowsException
2/21	20180221.01	Win10	The operation has been canceled	GetAsync_SetCookieContainer_CookieSent - link
2/25	20180225.01	Win10	The operation has been canceled	PostRewindableContentUsingAuth_NoPreAuthenticate_Success - link
2/25	20180225.01	Win7	The handle is invalid	HttpClientHandler_IncorrectCredentials_Fails - link
2/25	20180225.02	Win10	The handle is invalid	HttpClientHandler_Authentication_Succeeds - link
2/25	20180225.02	Win10	The operation has been canceled	PostUsingNoSpecifiedSemantics_UsesChunkedSemantics - link
2/26	20180226.01	Win8.1	The operation timed out	PostLongerContentLengths_UsesChunkedSemantics - link
2/26	20180226.02	Win10	The operation has been canceled	GetAsync_SetCookieContainer_CookieSent - link
2/26	20180226.02	Win10	The operation has been canceled	GetAsync_SetCookieContainer_CookieSent - link
2/26	20180226.02	Win8.1	The operation has been canceled	GetAsync_SupportedSSLVersion_Succeeds - link
2/28	20180228.07	Win7	The handle is invalid	ThresholdExceeded_ThrowsException
3/3	20180303.04	Win10	The operation has been canceled	GetAsync_SetCookieContainer_CookieSent
3/4	20180304.01	Win10	The operation has been canceled	GetAsync_SetCookieContainer_CookieSent
3/5	20180305.05	Win7	The operation has been canceled	ThresholdExceeded_ThrowsException
3/6	20180306.02	Win10	The operation has been canceled	GetAsync_SetCookieContainer_CookieSent
3/8	20180308.09	Win7	The handle is invalid	GetAsync_SetCookieContainer_CookieSent
3/11	20180311.02	Win8.1	The operation has been canceled	GetAsync_SetCookieContainer_CookieSent
3/22	20180322.01	Win10	The operation has been canceled	GetAsync_SetCookieContainer_CookieSent
3/22	20180322.06	Win7	The handle is invalid	ThresholdExceeded_ThrowsException
3/22	20180322.06	Win7	The operation has been canceled	SendAsync_SendSameRequestMultipleTimesDirectlyOnHandler_Success

[karelz]

@rmkerr this is 'misc' test failures which are similar to a few buckets you are already looking at ...

[rmkerr]

It looks to me like these can be traced back to #17542, since they all relate to WinHttp handles unexpectedly becoming invalid. So the failure happens when the http request is cancelled (probably via timeout?), and all of these failures are just how things fall apart after that. @davidsh you know a lot more about that issue, do you agree?

[davidsh]

It looks to me like these can be traced back to #17542, since they all relate to WinHttp handles unexpectedly becoming invalid. So the failure happens when the http request is cancelled (probably via timeout?), and all of these failures are just how things fall apart after that. @davidsh you know a lot more about that issue, do you agree?

Yes. The root cause is probably due to "timeouts/request cancellation". 9228 is a bug that when fixed will correct the error handling during cancellation so that the proper exception (OperationCanceled, etc.) will be thrown all the time instead of other random exceptions.

[karelz]

No luck in tracking it down now - see https://github.com/dotnet/corefx/issues/25880#issuecomment-360297737 for details, moving it to Future.

[joshfree]

latest failures

20180222.04 System.Net.Http.Functional.Tests Windows 10 Nano

20180222.04 System.Net.WebSockets.Client.Tests Ubuntu 14.04 / Windows 8 / Windows 7

[krwq]

Please paste if you got any more recent failures here as it seems helix has lost repros for anything before March 20. I'll try to get local repro for this for now

[karelz]

@krwq you can check Kusto. Moreover, you don't need recent repro to get Helix repro machine ;)

[krwq]

@karelz, just got machine from similar run, will be trying to get a repro now 😄

[krwq]

@karelz, I haven't a single repro on Win7 machine since I started running these tests in a loop 4h ago. The only repro I got was https://github.com/dotnet/corefx/issues/28696 but after locally disabling that test I got only clean runs (later even that stopped reproing for no reason). I'm not sure if it makes sense to continue this.. I'm trying to run this on manually created VM with only 2 cores and see if I'll be more lucky

[karelz]

The issue keeps happening in CI and @rmkerr was able to reproduce it on his machine in August-ish timeframe, so we need to find out how to catch this thing :)

[karelz]

@rmkerr is going to take a look ...