[release/6.0] add Http3LoopbackConnection.ShutdownAsync and use in AcceptConnectionAsync

[github-actions]

Backport of #58088 to release/6.0

Contributes to #57779

Test-only fixes. Should allow re-enabling a bunch of disabled tests (not part of this change).

Add a ShutdownAsync method on the HTTP3 loopback connection. This will send a GOAWAY frame to the client and wait for the client to close the connection gracefully. It will also eat connection abort exceptions if the client already has closed the connection.

Use this method in AcceptConnectionAsync, after the user's connection handling code has finished. The intent here is to avoid the kind of data loss seen in #57779. and avoid HTTP3-specific one-offs to reenable tests as in #58041.

@danmoseley

/cc @geoffkizer

Customer Impact

Testing

Risk

Low.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Backport of #58088 to release/6.0

/cc @geoffkizer

Customer Impact

Testing

Risk

Author:	github-actions[bot]
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[geoffkizer]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[karelz]

Triage: We want to add re-enabled tests to this PR to improve our ability of servicing 6.0.

[Anipik]

cc @danmoseley

[danmoseley]

Approved. Valuable test only changes.

[danmoseley]

@josalem BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover hung in coreclr windows x86 .
If you sum all the sleeps (assuming all retries are used) in Test_EventSource_EtwManifestGenerationRollover, they come to over 2 minutes. By default, RemoteExecutor times out after 60 seconds, as here. If you believe progress was happening it just didn't get enough time, you can increase that limit thus:
https://github.com/dotnet/runtime/blob/fc854d9921475366875658180afa495d452451be/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Unix.cs#L28-L27
My guess is that you expect it to take much less than 60 seconds though...?

[josalem]

My guess is that you expect it to take much less than 60 seconds though...?

I'm not actually familiar with this test. Looking at the failing test code, I would expect this to be making progress, so we can try to queue up a change as you suggested.

[danmoseley]

so we can try to queue up a change as you suggested.

Sounds good. it will log a complaint if 50% of the timeout is exhausted, which can suggest if the timeout is excessive (if it typically passes without the complaint, then it can probably be reduced). If you crank it up and do'nt see the warning, it can probably go down again some.

[ManickaP]

Triage: We want to add re-enabled tests to this PR to improve our ability of servicing 6.0.

I'm pushing cherry picked changes from these 7.0 issues: is:issue is:closed project:dotnet/runtime/2 label:disabled-test sort:updated-desc milestone:7.0.0

The changes are only re-enabling disabled tests to improve our test coverage in 6.0 branch.

@CarnaViire @wfurt @geoffkizer if I overlooked some PRs (re-enabling tests) that should go with this as well let me know.

@danmoseley does your approval still stand after we've added these tests?

cc @karelz

[ManickaP]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[karelz]

Thanks @ManickaP -- FYI @danmoseley this is what we planned to add to the PR before submitting it for approval - see my comment above #58336 (comment).

[karelz]

Test failures:

• 2x System.Net.Sockets.Tests.DnsEndPointTest.Socket_ConnectAsyncDnsEndPoint_HostNotFound on SLES.15
  • Unrelated, covered by Tests time out: System.Net.Sockets.Tests.DnsEndPointTest - mostly on SLES.15 in release/6.0 branches #57929
• 1x System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Cancellation_Test.Expect100Continue_WaitsExpectedPeriodOfTimeBeforeSendingContent on Ubuntu.1604

    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Cancellation_Test.Expect100Continue_WaitsExpectedPeriodOfTimeBeforeSendingContent [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.cs(111,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test.Expect100Continue_WaitsExpectedPeriodOfTimeBeforeSendingContent()
        --- End of stack trace from previous location ---

• 3x System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http3_Mock.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer on Debian.10, Fedora.34, SLES.15.
  • That may be related to [HTTP3] Loopback server WaitForCancellationAsync logic is wrong #58234 - do we need to disable it with the Loopback server changes?

    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http3_Mock.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs(33,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Finalization_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()
        --- End of stack trace from previous location ---

• runtime-stages - unrelated failures

Rerunning failed legs.

[karelz]

Re-run test failures:

• 1x System.Net.Http.Functional.Tests.SocketsHttpHandler_Http2KeepAlivePing_Test.KeepAlivePingDelay_Infinite_NoKeepAlivePingIsSent on Ubuntu.1604 (happened also in PR Remove two async state machines for typical HTTP/1.1 request path #58092 as the only 2 failures in last 2 months) -- likely unrelated flakiness

    System.Net.Http.Functional.Tests.SocketsHttpHandler_Http2KeepAlivePing_Test.KeepAlivePingDelay_Infinite_NoKeepAlivePingIsSent [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Http2KeepAlivePing.cs(45,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_Http2KeepAlivePing_Test.KeepAlivePingDelay_Infinite_NoKeepAlivePingIsSent()
        --- End of stack trace from previous location ---

• 5x Mock System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http3_Mock.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer on Fedora.34, Ubuntu.1804, Ubuntu.1604, CentOS.8, SLES.15 -- same as in previous run

      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs(33,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Finalization_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()
        --- End of stack trace from previous location ---

• 1x MsQuic System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http3_MsQuic.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer on Fedora.34

System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http3_MsQuic.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs(33,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Finalization_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()
        --- End of stack trace from previous location ---

Rerunning again

[karelz]

Re-run no.2 test failures:

• 1x System.Net.Sockets.Tests.DnsEndPointTest.Socket_ConnectAsyncDnsEndPoint_HostNotFound on SLES.15
  • Unrelated, covered by Tests time out: System.Net.Sockets.Tests.DnsEndPointTest - mostly on SLES.15 in release/6.0 branches #57929
• 4x System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http3_Mock.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer on Fedora.34, Ubuntu.1804, Ubuntu.1604, Debian.10

    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http3_Mock.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs(33,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Finalization_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()
        --- End of stack trace from previous location ---

[karelz]

Triage: Given the unexpected instability the test-only improvement PR brings on the one test above, we do not think it is worth putting it into 6.0. It would allows us to re-enable only 3 disabled tests, so no super-large value for 6.0.
Closing.

[karelz]

Note: 1 of the failures was actually on MsQuic variant of the test - see #58336 (comment).