Customizing HTTP/2 abort response

[JamesNK]

In gRPC there is the concept of a deadline. The deadline is sent by the client to the server with a gRPC call in a header. It is the client saying to the server "you have X seconds to finish this call, after-which give up".

• If the server finished the call in less than X sends then the response completes as usual.
• If the server exceeds the X seconds then it immediately finishes the HTTP/2 response, sending a RST_STREAM. The user's gRPC method code will likely still be executing after this. If they are good then it will be using a cancellation token, and they stop whatever they're doing quickly. But they might not, and continue doing stuff after the response is complete. This is fine. It is also fine if an error eventually gets thrown because they tried to do an HTTP thing.

Today the gRPC server is finishing the connection when the deadline is hit by calling HttpContext.Abort(). This immediately sends RST_STREAM with an INTERNAL_ERROR error code. The request delegate with the user's code is still running. Using Abort is not ideal, but its response seems to be acceptable to the various gRPC clients of the world.

Today that response on Abort looks very hard-coded in Http2Stream:

protected override void ApplicationAbort()
{
    var abortReason = new ConnectionAbortedException(CoreStrings.ConnectionAbortedByApplication);
    ResetAndAbort(abortReason, Http2ErrorCode.INTERNAL_ERROR);
}

I think the current abort behavior is fine for gRPC to use with deadline, the one thing that could be improved is the abort response over HTTP/2. We're sending INTERNAL_ERROR, and no status trailers. Another gRPC implementation I looked at immediately ends the response, but it sends RST_STREAM (NO_ERROR), along with some HTTP/2 trailing headers.

It would be nice to have a hook in ASP.NET Core to modify the response when an HTTP/2 abort occurs, such as changing the Http2ErrorCode used with RST_STREAM, or appending trailers.

Thoughts around the area?
Or alternatives?

[Tratcher]

it sends RST_STREAM (NO_ERROR), along with some HTTP/2 trailing headers.

How does it do that? It should only work if they send the trailers first as trailers after RST would be invalid.

It would be nice to have a hook in ASP.NET Core to modify the response when an HTTP/2 abort occurs, such as changing the Http2ErrorCode used with RST_STREAM, or appending trailers.

I could image having an HTTP Feature and response extension for providing a Reset method with a custom error code (and another one for GoAway). Also including trailers would be complicated.

You still have a multi-threading race condition here though. Only one thread is supposed to touch HttpContext, so calling Reset from a timer thread would not be safe.

[JamesNK]

How does it do that? It should only work if they send the trailers first as trailers after RST would be invalid.

HEADER frame first, then RST_STREAM

Only one thread is supposed to touch HttpContext

Does that include HttpContext.Abort? I don't mind if regular ASP.NET Core thread throws an error because abort has been called. How would you suggest implementing deadline functionality that I've described without another thread?

[Tratcher]

Does that include HttpContext.Abort? I don't mind if regular ASP.NET Core thread throws an error because abort has been called. How would you suggest implementing deadline functionality that I've described without another thread?

Yes, right now it does. Let's discuss the thread safety issue over on #9239 (comment).

[halter73]

It would be nice to have a hook in ASP.NET Core to modify the response when an HTTP/2 abort occurs, such as changing the Http2ErrorCode used with RST_STREAM, or appending trailers.

Could we just add a new HTTP/2-specific feature that allows you to both abort the request and configure RST_STREAM all at once? Basically it would be a specially HTTP/2 version of support with extra parameters. Since you already control the timer callback that calls IHttpRequestLifetimeFeature.Abort(), I don't see the need for a callback-based API.

[JamesNK]

IHttp2AbortThingyFeature.Abort(Http2ErrorCode) could be enough. I want to double check whether returning trailers with a gRPC deadline is a nice to have or not.

[Tratcher]

RE: Trailers, sending them would be a protocol violation if the response had a content-length and you hadn't already finished the body.

[JamesNK]

Ok, I've refactored gRPC deadline logic so that it no longer uses a Timer plus HttpContext.Abort

PR here - https://github.com/grpc/grpc-dotnet/pull/301/files

There is now a separate path the server uses when there is a deadline. It produces a better response than HttpContext.Abort, but it allocates a lot more.

• Before there was just a Timer. Allocations were 520B.
• With refactor there is a CTS, Task.Delay, Task.WhenAny. Allocations are 1.09KB

Deadline related code here:
https://github.com/grpc/grpc-dotnet/blob/e2391970cce9b8428a1dc30ea8e612d617bd4e02/src/Grpc.AspNetCore.Server/Internal/CallHandlers/ServerCallHandlerBase.cs#L99-L145

I do not know whether this a good idea. It would be great to have input from experts in this area.

// @davidfowl @Tratcher @halter73

[JamesNK]

I don't think my refactor in gRPC will work. It results in holding onto the HttpContext/Http2Stream after the request pipeline has finished. Calling HttpContext.Abort is not a solution because trailers need to be written and they aren't when the request is aborted.

I think the solution is some changes in Kestrel and this issue's original request is the best way forward I can think of:

• Add an option to customize the Http2ErrorCode that is sent on Abort
• Add an option to write trailers on Abort (or maybe always writing trailers on abort makes sense in HTTP/2)

Good idea? Bad idea?
Simple to do? Hard?

[JamesNK]

Alternatively, an option could be to use Response.BodyWriter.Complete() to send the trailers and RST_STREAM

#7370

[halter73]

If BodyWriter.Complete() was used to send the trailers, that would result in an a frame with an end-of-stream flag. I'm not sure it's valid to send an RST stream after that.

Add an option to customize the Http2ErrorCode that is sent on Abort

This seems necessary.

Add an option to write trailers on Abort (or maybe always writing trailers on abort makes sense in HTTP/2)

@Tratcher and I were discussing this yesterday. This seems weird to us. What's the other gRPC implementation that does this?

[JamesNK]

If BodyWriter.Complete() was used to send the trailers, that would result in an a frame with an end-of-stream flag. I'm not sure it's valid to send an RST stream after that.

It is valid.

An HTTP response is complete after the server sends — or the client receives — a frame with the END_STREAM flag set (including any CONTINUATION frames needed to complete a header block). A server can send a complete response prior to the client sending an entire request if the response does not depend on any portion of the request that has not been sent and received. When this is true, a server MAY request that the client abort transmission of a request without error by sending a RST_STREAM with an error code of NO_ERROR after sending a complete response (i.e., a frame with the END_STREAM flag). Clients MUST NOT discard responses as a result of receiving such a RST_STREAM, though clients can always discard responses at their discretion for other reasons.

https://httpwg.org/specs/rfc7540.html#rfc.section.8.1

What's the other gRPC implementation that does this?

The primary gRPC implementation (C-core) does this. In this Wireshark trace of the deadline response from C-core you can see trailers followed by a RST_STREAM