Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP/2 stress server intermitently fails in duplex scenarios #42200

Closed
ManickaP opened this issue Sep 14, 2020 · 1 comment · Fixed by #54552
Closed

HTTP/2 stress server intermitently fails in duplex scenarios #42200

ManickaP opened this issue Sep 14, 2020 · 1 comment · Fixed by #54552
Assignees
@alnikola
Labels
area-System.Net.Http bug tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Milestone
6.0.0

Comments

@ManickaP
Copy link
Member

When sending custom content in H/2 and using simple echo server. Then client disposes the response which might send RESET before the last DATA frame with END_STREAM flag leading to a server-side exception:

System.IO.IOException: The client reset the request stream.
  at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)

The root of the issue resides in sending END_STREAM completely asynchronously.

  • In case when our own predefined contents are used, this frame is send asynchronously and might be sent after SendAsync has finished.
  • In case custom content is used, the end of its SerializeToStreamAsync is not a strong indicator that the END_STREAM has been sent, it still happens completely asynchronously.

Relates to #1511
@ManickaP ManickaP added this to the 6.0.0 milestone Sep 14, 2020
@ghost
Copy link

ghost commented Sep 14, 2020

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Sep 14, 2020
@ManickaP ManickaP removed the untriaged New issue has not been triaged by the area owner label Sep 14, 2020
@ManickaP ManickaP mentioned this issue Sep 14, 2020
Open
@karelz karelz added the bug label Sep 15, 2020
@ManickaP ManickaP mentioned this issue Sep 18, 2020
Closed
@karelz karelz added the tenet-reliability Reliability/stability related issue (stress, load problems, etc.) label May 6, 2021
@alnikola alnikola self-assigned this Jun 21, 2021
This was referenced Jun 21, 2021
Closed
Merged
@ghost ghost added in-pr There is an active PR which will close this issue when it is merged and removed in-pr There is an active PR which will close this issue when it is merged labels Jun 22, 2021
alnikola added a commit that referenced this issue Jun 29, 2021
@alnikola
Avoid sending EndStream after RST_STREAM with dedicated lock (#54552)
5d29560 
A race condition between sending RST_STREAM and checking conditions for sending EndStream was discovered during stress testing. It happens to be possible that in time after Http2Stream [checked the _responseCompletionState](https://github.com/dotnet/runtime/blob/183c4d100f68fb6c177a1fe71809d581aa25e47b/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs#L275) and [actually send EndStream](https://github.com/dotnet/runtime/blob/183c4d100f68fb6c177a1fe71809d581aa25e47b/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs#L288), a concurrent call to [Cancel method sends a RST_STREAM frame](https://github.com/dotnet/runtime/blob/183c4d100f68fb6c177a1fe71809d581aa25e47b/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs#L389). Such reordering is disallowed by HTTP/2 protocol.

Note: The issue and fix were verified manually under the debugger because currently it's not clear how to reliably simulate that situation.

Fixes #42200
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jun 29, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Jul 29, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@alnikola alnikola

Labels
area-System.Net.Http bug tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Projects
None yet
Milestone
6.0.0
Development

Successfully merging a pull request may close this issue.

Avoid sending EndStream after RST_STREAM with dedicated lock alnikola/runtime
4 participants
@karelz @ManickaP @Dotnet-GitSync-Bot @alnikola