GatewayStoreClient buffers entire response body and makes another copy of it for later use

[magole]

Describe the bug
GatewayStoreClient buffers entire response body and makes another copy of it for later use. With large number of docs in the response, this doesn't scale well on a busy server.

To Reproduce

var client = new HttpClient()
{
    MaxResponseContentBufferSize = 64 * 1024, // 64 KiB
};

Expected behavior
Cosmos client should use streaming and never buffer entire response in memory.
Cosmos client should always use HttpCompletionOption.ResponseHeadersRead for all HTTP calls.

Actual behavior

Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: InternalServerError (500); Substatus: 0; ActivityId: 00000000-0000-0000-0000-000000000000; Reason: (Cannot write more bytes to the buffer than the configured maximum buffer size: 65536.);
 ---> System.Net.Http.HttpRequestException: Cannot write more bytes to the buffer than the configured maximum buffer size: 65536.
   at System.Net.Http.HttpContent.LimitMemoryStream.CheckSize(Int32 countToAdd)
   at System.Net.Http.HttpContent.LimitMemoryStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore(Stream destination, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

Environment summary
Linux, NET Core 3.1

Additional context
https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/GatewayStoreClient.cs

Line 369

return this.httpClient.SendHttpAsync(
                () => this.PrepareRequestMessageAsync(request, physicalAddress),
                resourceType,
                diagnosticsContext,
                cancellationToken);

https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/HttpClient/CosmosHttpClientCore.cs

Line 201

        public override Task<HttpResponseMessage> SendHttpAsync(
            Func<ValueTask<HttpRequestMessage>> createRequestMessageAsync,
            ResourceType resourceType,
            CosmosDiagnosticsContext diagnosticsContext,
            CancellationToken cancellationToken)
        {
            return this.SendHttpAsync(
                createRequestMessageAsync,
                HttpCompletionOption.ResponseContentRead,
                resourceType,
                diagnosticsContext,
                cancellationToken);
        }

[j82w]

I agree that it is an issue, but we mainly optimize for Direct mode as it is the recommend mode for performance scenarios. Is there a reason you are using gateway mode?

[magole]

We run on Kubernetes and have limited number of connections for containers. With large number of pods on each node and Azure limits per node we quickly run out of connections. We tried Direct mode and had to revert back to gateway mode. Also using Gateway mode provides better insights/logs for HTTP calls out of the box.

[ealsur]

Question: Have you tried the reducing the connections with PortReuseMode and IdleConnectionTimeout? That would reduce connections when you have sporadic usage.

[j82w]

@magole please try the latest 3.13.0

[ghost]

Closing due to in-activity, pease feel free to re-open.