Random Sent 0 request content bytes error at POST requests

[WhitePhoera]

Describe the bug

After updating proxy with YARP 2.0.0 some users started to complain about 502 error.
After enabling logs i found that this is

info: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
System.Net.Http.HttpRequestException: Sent 0 request content bytes, but Content-Length promised 526.
at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)

Request finished HTTP/2 POST <some_url> multipart/form-data;+boundary=----WebKitFormBoundaryhPCLMtBnddLba95L 526 - 502 - - 1.5743ms

probably releated to #1657, there is comment that someone also have same problem after updating.

To Reproduce

I was not able to manually reproduce it, so not sure what how it exactly happens.

Further technical details

Linux
Asp.Net Core 7.0.2
.Net 7
YARP 2.0.0
Kestrel

My thought was that i have conflicts with PassthoughtStream i made.

        reqBody = new PassthoughtStream(context.Request.Body);
        respBody = new PassthoughtStream(context.Response.Body);

but manual tests shows nothing.
that wrapper just counts read and writen bytes and proxy all calls to original stream(which i restore ofc)
i use them to count passed bytes.

any tip on the issue?

[Tratcher]

Can you share the code for PassthoughtStream?

[WhitePhoera]

PassthoughtStream

    public class PassthoughtStream : Stream
    {
        public PassthoughtStream(Stream originalStream)
        {
            OriginalStream = originalStream ?? throw new ArgumentNullException(nameof(originalStream));
        }

        public long ReadBytes { get; set; } = 0;
        public long WritenBytes { get; set; } = 0;
        public Stream OriginalStream { get; }

        public override bool CanRead => OriginalStream.CanRead;

        public override bool CanSeek => OriginalStream.CanSeek;

        public override bool CanTimeout => OriginalStream.CanTimeout;

        public override bool CanWrite => OriginalStream.CanWrite;

        public override long Length => OriginalStream.Length;

        public override long Position { get => OriginalStream.Position; set => OriginalStream.Position = value; }
        public override int ReadTimeout { get => OriginalStream.ReadTimeout; set => OriginalStream.ReadTimeout = value; }
        public override int WriteTimeout { get => OriginalStream.WriteTimeout; set => OriginalStream.WriteTimeout = value; }






        public override bool Equals(object? obj)
        {
            return OriginalStream.Equals(obj);
        }

        public override void Flush()
        {
            OriginalStream.Flush();
        }

        public override Task FlushAsync(CancellationToken cancellationToken)
        {
            return OriginalStream.FlushAsync(cancellationToken);
        }
        protected override void Dispose(bool disposing)
        {
        }
        public override int GetHashCode()
        {
            return OriginalStream.GetHashCode();
        }

        public override int Read(byte[] buffer, int offset, int count)
        {
            int res = OriginalStream.Read(buffer, offset, count);
            ReadBytes += res;
            return res;
        }

        public override int Read(Span<byte> buffer)
        {
            int res = OriginalStream.Read(buffer);
            ReadBytes += res;
            return res;
        }

        public override async Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
        {
            int res = await OriginalStream.ReadAsync(buffer, offset, count, cancellationToken);
            ReadBytes += res;
            return res;
        }

        public override async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
        {
            int res= await OriginalStream.ReadAsync(buffer, cancellationToken);
            ReadBytes += res;
            return res;
        }

        public override int ReadByte()
        {
            int res= OriginalStream.ReadByte();
            if (res >= 0)
                ReadBytes++;
            return res;
        }

        public override long Seek(long offset, SeekOrigin origin)
        {
            return OriginalStream.Seek(offset, origin);
        }

        public override void SetLength(long value)
        {
            OriginalStream.SetLength(value);
        }

        public override string? ToString()
        {
            return OriginalStream.ToString();
        }

        public override void Write(byte[] buffer, int offset, int count)
        {
            OriginalStream.Write(buffer, offset, count);
            WritenBytes += count;
        }

        public override void Write(ReadOnlySpan<byte> buffer)
        {
            OriginalStream.Write(buffer);
            WritenBytes += buffer.Length;
        }

        public override async Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
        {
            await OriginalStream.WriteAsync(buffer, offset, count, cancellationToken);
            WritenBytes += count;
        }

        public override async ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken = default)
        {
            await OriginalStream.WriteAsync(buffer, cancellationToken);
            WritenBytes+= buffer.Length;
        }

        public override void WriteByte(byte value)
        {
            OriginalStream.WriteByte(value);
            WritenBytes++;
        }

    }

and i replace it:

        var reqBody = new PassthoughtStream(context.Request.Body);
        var respBody = new PassthoughtStream(context.Response.Body);
        context.Response.Body = respBody;
        context.Request.Body = reqBody;

i downgraded to 1.1.0 and same issue does not appear.

[Tratcher]

That seems harmless enough. We'll have to investigate further. Let us know if you narrow down the repro any.

[benjaminpetit]

It might be related to #2070, but it is hard to check without a repro.

PassthoughtStream seems fine to me, but it might be worth it to check without it, if possible?

[benjaminpetit]

Could you try with the latest daily build that should include #2119 and see if your issue is fixed?

[karelz]

@WhitePhoera will you be able to check the daily builds to see if it fully addresses your problem?
It would be great help for us. Thanks!

[karelz]

Triage: No response, we assume everything is fine and the fix helped. Closing.
If there is evidence of problems, feel free to reopen or file a new issue. Thanks!

[WhitePhoera]

@WhitePhoera will you be able to check the daily builds to see if it fully addresses your problem? It would be great help for us. Thanks!

Sorry, i was distracted, sadly no, it happened on production server with angry client, so i can't risk to test it in live.
I probably will wait for stable release with fix and test it with help of that client.

since that can be related to some individual network settings(since i was not able to repeat it myself, even on same server)

[arnonax-tr]

I ran into the same problem and I believe that I found the cause, and also managed to resolve it: I have a Request Transform that reads data from the body stream. This advanced the stream, so when the forwarder tried to read from it started from the end... The solution was to add the following line after reading from the body:

requestContext.HttpContext.Request.Body.Seek(0, SeekOrigin.Begin);

However, I think that it would be better if this happened automatically after reverse-proxy calls the request transformers, and before forwarding the request to the destination.

[Tratcher]

@arnonax-tr The request body isn't even seekable/buffered by default. Any component that enables buffering and reads from the stream is responsible for resetting it afterwards.