Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure: System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.ServerClosesOutboundControlStream_ClientClosesConnection - Status code: Unknown (0xdde6f334) #75270

Closed
karelz opened this issue Sep 8, 2022 · 9 comments
Closed

Test failure: System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.ServerClosesOutboundControlStream_ClientClosesConnection - Status code: Unknown (0xdde6f334) #75270

karelz opened this issue Sep 8, 2022 · 9 comments
Labels
area-System.Net.Quic bug test-failure test-run-core Test failures in .NET Core test runs
Milestone
9.0.0

Comments

@karelz
Copy link
Member

karelz commented Sep 8, 2022 •
edited
Loading

Affected test:

  • ServerClosesOutboundControlStream_ClientClosesConnection - System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.ServerClosesOutboundControlStream_ClientClosesConnection
Day Run OS Details
9/6 Rolling run 6173 net7.0-Linux-Release-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open Console log

I couldn't find more occurrences on 9/8 when looking at last 7 days in Runfo (they have different symptoms).

Error - Console log:

    System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.ServerClosesOutboundControlStream_ClientClosesConnection(graceful: False) [FAIL]
      Assert.IsType() Failure
      Expected: System.Net.Http.HttpProtocolException
      Actual:   System.Net.Quic.QuicException
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http3.cs(39,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.AssertProtocolErrorAsync(Int64 errorCode, Func`1 task)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http3.cs(1640,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.<>c__DisplayClass39_0.<<ServerClosesOutboundControlStream_ClientClosesConnection>b__1>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(128,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(92,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(55,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http3.cs(1644,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.ServerClosesOutboundControlStream_ClientClosesConnection(Boolean graceful)
        --- End of stack trace from previous location ---
      Output:
        System.Net.Http.HttpRequestException: An error occurred while sending the request.
         ---> System.Net.Quic.QuicException: An internal error has occurred. Status code: Unknown (0xdde6f334)
           at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
           at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 616
           at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 653
           at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 677
        --- End of stack trace from previous location ---
           at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 273
           at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 309
           at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 837
           at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 356
           at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 323
           --- End of inner exception stack trace ---
           at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 323
           at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 323
           at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 252
           at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 252
           at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1004
           at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1160
           at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 85
           at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 558
           at Xunit.Assert.RecordExceptionAsync(Func`1 testCode) in /_/src/xunit.assert/Asserts/Record.cs:line 111
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Sep 8, 2022
@ghost
Copy link

ghost commented Sep 8, 2022

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details
Day Run OS Details
9/6 Rolling run 6173 net7.0-Linux-Release-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open Console log

I couldn't find more occurrences on 9/8 when looking at last 7 days in Runfo (there different symptoms).

Error - Console log:

    System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.ServerClosesOutboundControlStream_ClientClosesConnection(graceful: False) [FAIL]
      Assert.IsType() Failure
      Expected: System.Net.Http.HttpProtocolException
      Actual:   System.Net.Quic.QuicException
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http3.cs(39,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.AssertProtocolErrorAsync(Int64 errorCode, Func`1 task)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http3.cs(1640,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.<>c__DisplayClass39_0.<<ServerClosesOutboundControlStream_ClientClosesConnection>b__1>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(128,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(92,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(55,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http3.cs(1644,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.ServerClosesOutboundControlStream_ClientClosesConnection(Boolean graceful)
        --- End of stack trace from previous location ---
      Output:
        System.Net.Http.HttpRequestException: An error occurred while sending the request.
         ---> System.Net.Quic.QuicException: An internal error has occurred. Status code: Unknown (0xdde6f334)
           at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
           at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 616
           at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 653
           at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 677
        --- End of stack trace from previous location ---
           at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 273
           at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 309
           at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 837
           at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 356
           at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 323
           --- End of inner exception stack trace ---
           at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 323
           at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 323
           at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 252
           at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 252
           at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1004
           at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1160
           at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 85
           at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 558
           at Xunit.Assert.RecordExceptionAsync(Func`1 testCode) in /_/src/xunit.assert/Asserts/Record.cs:line 111
Author: karelz
Assignees: -
Labels:

area-System.Net.Quic
Milestone: -

@karelz
Copy link
Member Author

karelz commented Sep 27, 2022

Triage: After @antonfirsov's changes we should not propagate QuicExeption -- this is weird, we should take a look.

@karelz karelz added bug and removed untriaged New issue has not been triaged by the area owner labels Sep 27, 2022
@karelz karelz added this to the 8.0.0 milestone Sep 27, 2022
@rzikm
Copy link
Member

rzikm commented Sep 29, 2022

0xdde6f334 does not seem to be any QUIC Status code used in Linux, so this is highly suspicious.

@wfurt
Copy link
Member

wfurt commented Sep 29, 2022

are there cases when the status is not errno @nibanks?
Of course if any of the pointers is bogus we would parse invalid memory.

@nibanks
Copy link

nibanks commented Sep 29, 2022

I don't recognize the status code. Is this coming from stream shutdown event? If so, that means it's an application layer error code, not a platform/QUIC one.

@rzikm
Copy link
Member

rzikm commented Sep 30, 2022

I don't recognize the status code. Is this coming from stream shutdown event? If so, that means it's an application layer error code, not a platform/QUIC one.

We don't use such values in our tests.

Based on the callstack, the value is definitely coming from SHUTDOWN_COMPLETE.ConnectionCloseStatus (for app closure we throw different exception and the app error code comes from SHUTDOWN_COMPLETE.ConnectionErrorCode field).

It could mean that we still have some memory corruption going on :/

@ManickaP
Copy link
Member

Triage: check occurrence, close if not happening anymore.

@rzikm
Copy link
Member

rzikm commented Jul 20, 2023

Happens occasionally, and passes on reruns. none of the recent failures had any weird weird MsQuic return code. I think we can close this safely.

@rzikm rzikm removed their assignment Jul 20, 2023
@karelz karelz modified the milestones: 8.0.0, 9.0.0 Aug 14, 2023
@karelz karelz added the test-run-core Test failures in .NET Core test runs label Sep 21, 2023
@liveans
Copy link
Member

liveans commented Oct 4, 2023

No failure and pass on rerun after #90253 merged (1 month), closing this, we can re-open it if we see any occurrence

@liveans liveans closed this as completed Oct 4, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Nov 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Net.Quic bug test-failure test-run-core Test failures in .NET Core test runs
Projects
None yet
Milestone
9.0.0
Development

No branches or pull requests
6 participants
@karelz @ManickaP @wfurt @nibanks @rzikm @liveans