Failures in SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3 test

[jkotas]

Hit in #74885

Full log https://dev.azure.com/dnceng/public/_build/results?buildId=1979617&view=ms.vss-test-web.build-test-results-tab&runId=50623688&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=203530

System.AggregateException : One or more errors occurred. (Assert.Contains() Failure
Not found: 1024
In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Quic.QuicException: An internal error has occurred. Status code: QUIC_STATUS_SUCCESS
at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
at System.Net.Quic.QuicStream.HandleEventShutdownComplete(SHUTDOWN_COMPLETE_e__Struct& data) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 595
at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 632
at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 656
--- End of stack trace from previous location ---
at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 235
at System.Net.Quic.QuicStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 287 at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 788 at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 337 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206 --- End of inner exception stack trace --- at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 308 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 237 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 251 at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 990 at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1034 at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30 at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 532 at Xunit.Assert.RecordExceptionAsync(Func1 testCode) in //src/xunit.assert/Asserts/Record.cs:line 111) (An internal error has occurred. Status code: QUIC_STATUS_SUCCESS)

{ "ErrorMessage":"System.Net.Quic.QuicException: An internal error has occurred. Status code: QUIC_STATUS_SUCCESS" } 

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hit in #74885

Full log https://dev.azure.com/dnceng/public/_build/results?buildId=1979617&view=ms.vss-test-web.build-test-results-tab&runId=50623688&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=203530

System.AggregateException : One or more errors occurred. (Assert.Contains() Failure
Not found: 1024
In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Quic.QuicException: An internal error has occurred. Status code: QUIC_STATUS_SUCCESS
at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
at System.Net.Quic.QuicStream.HandleEventShutdownComplete(SHUTDOWN_COMPLETE_e__Struct& data) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 595
at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 632
at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 656
--- End of stack trace from previous location ---
at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 235
at System.Net.Quic.QuicStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 287 at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 788 at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 337 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206 --- End of inner exception stack trace --- at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 308 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 237 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 251 at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 990 at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1034 at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30 at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 532 at Xunit.Assert.RecordExceptionAsync(Func1 testCode) in //src/xunit.assert/Asserts/Record.cs:line 111) (An internal error has occurred. Status code: QUIC_STATUS_SUCCESS)

Author:	jkotas
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hit in #74885

Full log https://dev.azure.com/dnceng/public/_build/results?buildId=1979617&view=ms.vss-test-web.build-test-results-tab&runId=50623688&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=203530

System.AggregateException : One or more errors occurred. (Assert.Contains() Failure
Not found: 1024
In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Quic.QuicException: An internal error has occurred. Status code: QUIC_STATUS_SUCCESS
at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
at System.Net.Quic.QuicStream.HandleEventShutdownComplete(SHUTDOWN_COMPLETE_e__Struct& data) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 595
at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 632
at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in //src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 656
--- End of stack trace from previous location ---
at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in //src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 235
at System.Net.Quic.QuicStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 287 at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 788 at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 337 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206 --- End of inner exception stack trace --- at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 308 at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 237 at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 251 at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 990 at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1034 at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30 at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 532 at Xunit.Assert.RecordExceptionAsync(Func1 testCode) in //src/xunit.assert/Asserts/Record.cs:line 111) (An internal error has occurred. Status code: QUIC_STATUS_SUCCESS)

Author:	jkotas
Assignees:	-
Labels:	area-System.Net.Http, area-System.Net.Quic
Milestone:	-

[wfurt]

may be fixed by #74634 (and #74611 it depends on)
cc: @rzikm

In either case it feels strange to throw with QUIC_STATUS_SUCCESS....

[rzikm]

Yes, the thing is... this is the code we receive from MsQuic... so if this continues after we merge the two PRs you mention (and we rule out any memory corruption done by us), we would need to involve MsQuic team.

[CarnaViire]

Also hit in #74669 (log) and #74645 (log) with a bit different error:

System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException(maxResponseHeadersLength: 1) [FAIL]
      System.AggregateException : One or more errors occurred. (Assert.Contains() Failure
      Not found: 1024
      In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request.
                  ---> System.Net.Quic.QuicException: The connection timed out from inactivity.

I think we've already seen this type of failure before, when @MihaZupan just added it?

PR to disable: #75149

[MihaZupan]

Yes, this test was already disabled in #73945 and then re-enabled in #74479.
The fact that it's still failing indicates a QUIC/HTTP3 issue.

[CarnaViire]

@MihaZupan I meant specifically a "timeout from inactivity". Because there were a bunch of different failures as well

[ManickaP]

No hit in a month, last comments from sept, closing.

[jkotas]

The test disabled against this issue needs to be re-enabled before closing:

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs

Line 1607 in 7bcd509

[ActiveIssue("https://github.com/dotnet/runtime/issues/74896")]

[ManickaP]

The test disabled against this issue needs to be re-enabled before closing:

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs

Line 1607 in 7bcd509

[ActiveIssue("https://github.com/dotnet/runtime/issues/74896")]

🤦 Thanks for catching that, I completely overlooked it.

[karelz]

The test was re-enabled in PR #91629 on 9/6.

The same test started failing with different error on 9/7 - reported in #91757
The test was disabled against THIS issue in PR #91930 on 9/12. That was an accident -- we should disable it against the new issue #91757 which captures latest failure.

@liveans please change the test number against which it is disabled, then let's close this issue as dupe of #91757 (while removing the disabled-test label). Let's keep it in 9.0.

[liveans]

I changed the issue number on the test, closing this issue as a duplicate of #91757, and deleting the disabled-test label.

[karelz]

Duplicate of #91757