HttpForwarder failed to send body after update to version 1.1.0-rc.1.22152.1 from 1.0.1

[illay1994]

Describe the bug

We have simple yarp proxy with configuration:

        services.AddReverseProxy().LoadFromConfig(Configuration.GetSection("ReverseProxy"));

and

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapReverseProxy();
        });

config file:

 "ReverseProxy": {
    "Routes": {
      "route": {
        "ClusterId": "cluster",
        "AuthorizationPolicy": "policy",
        "Match": {
          "Path": "/v1/{**remainder}"
        }
      }
    },
    "Clusters": {
      "cluster": {
        "Destinations": {
          "api": {
            "Address": "http://localhost:60825/subpath1/subpath2"
          }
        }
      }
    }
  }

To Reproduce

in version 1.0.1 all working fine

after update to version 1.1.0-rc.1.22152.1 get error in log and return 400 HttpError code

logs:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 POST http://192.168.0.2:5000/v1/api application/json;+charset=UTF-8 36
----
internal Authentication logs 
----
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'route'
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[9]
      Proxying to http://localhost:60825/subpath1/subpath2/v1/api HTTP/2 RequestVersionOrLower no-streaming
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      RequestBodyClient: The client reported an error when copying the request body.
      System.AggregateException: One or more errors occurred. (Sent 0 request content bytes, but Content-Length promised 36.) (Error while copying content to a stream.)
       ---> System.InvalidOperationException: Sent 0 request content bytes, but Content-Length promised 36.
         --- End of inner exception stack trace ---
       ---> (Inner Exception #1) System.Net.Http.HttpRequestException: Error while copying content to a stream.
       ---> System.IO.IOException: An error occurred when reading the request body from the client.
       ---> System.InvalidOperationException: Sent 0 request content bytes, but Content-Length promised 36.
         --- End of inner exception stack trace ---
         at Yarp.ReverseProxy.Forwarder.StreamCopyHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
         at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
         at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer)<---

info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'route'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 POST http://192.168.0.2:5000/v1/api application/json;+charset=UTF-8 36 - 400 0 - 168.2274ms

Further technical details

• Include the version of the packages you are using 1.1.0-rc.1.22152.1
• net6.0
• Windows 10 and docker image ( mcr.microsoft.com/dotnet/aspnet:6.0)

[MihaZupan]

Sent 0 request content bytes, but Content-Length promised 36.

The error indicates that YARP wasn't able to read any bytes from the request body.
Are you modifying/reading the body in code at any point?

[illay1994]

Hi MihaZupan

No.

Client not support http2.
Client still used http1.1

[MihaZupan]

Were these requests that were previously succeeding?

It might be that this is a case where the client disconnected before sending the request content.
Previously this may have shown up as some sort of cancellation exception, but now it's an input error.

[illay1994]

Hi MihaZupan

It is reproduces always.

It is also reproduceble from swagger or curl

[MihaZupan]

Can you provide a minimal runnable repro we could try?

[illay1994]

It is start reproducing with after merge of this PR
#1415

[illay1994]

If you call in any middleware before UseEndpoints

 app.Use((context, next) =>
        {
            context.Request.EnableBuffering();
            ....
            return next(context);
        });

Proxy will fail

[Tratcher]

What's the rest of that middleware look like? Are you remembering to re-wind the request body after you read it?

[illay1994]

just this without extra lines

 app.Use((context, next) =>
        {
            context.Request.EnableBuffering();
            return next(context);
        });

I am planning to delete this middleware.

[karelz]

Triage: @illay1994 can you please post a minimal repro (app) for us to check? The code snippets above are not fully helping :(
Thanks!

[illay1994]

Code exampe:

illay1994@817288d

test url:

curl -X POST "https://localhost:5001/post" -H "accept: application/json" -d '{"a":123}'

[Tratcher]

Thanks, I was able to reproduce this and track it back to dotnet/aspnetcore#41287. EnableBuffering is not compatible with the zero-byte read pattern, that's something we'll need to fix in aspnetcore.

For Triage: recommend closing this as external.

[Tratcher]

Related: #1665.

Updated proposal: Skip zero-byte reads for 3.1 and 5.0, and patch AspNetCore 6 for any cases we find.

[karelz]

Fixed by #1666

[0xStuart]

I am using yarp to proxy Grafana but I keep getting this error with some of the content.
System.Net.Http.HttpRequestException: Sent 0 request content bytes, but Content-Length promised 493.

Googling the issue brought me here. .Net 7.0.3 and Yarp 2.0 So shouldn't be getting this issue. Any ideas?

[johnkors]

@karelz, this seems to still be an issue. I'm unable to read the request body.

Using a similar approach as OP:

app.UseSerilogRequestLogging(options =>
{
    options.EnrichDiagnosticContext = (diagnosticContext, httpContext) =>
    {
        var body = GetRawBodyAsync(httpContext.Request).GetAwaiter().GetResult();
        diagnosticContext.Set("RequestBody", body);
    };
    
    async Task<string> GetRawBodyAsync(HttpRequest request)
    {
        request.EnableBuffering();
        using var reader = new StreamReader(request.Body, Encoding.UTF8);
        var body = await reader.ReadToEndAsync();
        request.Body.Position = 0;
        return body;
    }
});

app.MapReverseProxy(); // 👈 Filtered on certain endpoints & cannot log response body
app.Map("/{**catchall}", Results.BadRequest); // logs response body just fine

Using

<PackageReference Include="Serilog.AspNetCore" Version="7.0.0" />
<PackageReference Include="Yarp.ReverseProxy" Version="2.0.1" />

[Tratcher]

Comments on closed issues are not tracked, please open a new issue with the details for your scenario.