GetFromJsonAsAsyncEnumerable not working when using custom httpHanlder with compression

[WeihanLi]

Description

Trying to produce server stream response, happen to find that when I use AutomaticDecompression option, the response is different when not using the option

when using the AutomaticDecompression option seemed the response is consumed at the end of the response, is this by design?

Reproduction Steps

var url = "http://localhost:5000/api/Values";
{
    Console.WriteLine($"Send request at {DateTimeOffset.Now}");
    using var httpClient = new HttpClient();
    await foreach (var (key, value) in httpClient.GetFromJsonAsAsyncEnumerable<KeyValuePair<string, string>>(url))
    {
        Console.WriteLine($"Received response at {DateTimeOffset.Now} : {key} {value}");
    }
}

Console.WriteLine(new string('-', 50));

{
    Console.WriteLine($"Send request at {DateTimeOffset.Now}");
    using var httpClient = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.All });
    await foreach (var (key, value) in httpClient.GetFromJsonAsAsyncEnumerable<KeyValuePair<string, string>>(url))
    {
        Console.WriteLine($"Received response at {DateTimeOffset.Now} : {key} {value}");
    }
}

output

The server is producing three items with two seconds delay in between

Expected behavior

The response should remain the same behavior

Actual behavior

The response behavior differs

Regression?

No response

Known Workarounds

No response

Configuration

The sample app is targeted for net8.0

dotnet --info output

.NET SDK:
 Version:           9.0.100-preview.2.24157.14
 Commit:            f7466905f9
 Workload version:  9.0.100-manifests.04914b26
 MSBuild version:   17.10.0-preview-24127-03+6f44380e4

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19045
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      9.0.0-preview.2.24128.5
  Architecture: x64
  Commit:       8e5e748c5c

.NET SDKs installed:
  2.2.207 [C:\Program Files\dotnet\sdk]
  3.1.426 [C:\Program Files\dotnet\sdk]
  5.0.403 [C:\Program Files\dotnet\sdk]
  5.0.408 [C:\Program Files\dotnet\sdk]
  6.0.100 [C:\Program Files\dotnet\sdk]
  6.0.129 [C:\Program Files\dotnet\sdk]
  6.0.202 [C:\Program Files\dotnet\sdk]
  6.0.421 [C:\Program Files\dotnet\sdk]
  7.0.118 [C:\Program Files\dotnet\sdk]
  8.0.104 [C:\Program Files\dotnet\sdk]
  9.0.100-preview.2.24157.14 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0-preview.2.24128.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0-preview.2.24128.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.0-preview.2.24128.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Other information

No response

[stephentoub]

What server are you connecting to? Are you certain the server is putting the data on the wire as quickly as you expect? Do you have an endpoint we can try or a repro for the server side to match?

[WeihanLi]

@stephentoub thanks very much for the quick reply, I'm using the aspnet kestrel server, it could relate to the server response compression, working without the response compression

I add a sample to reproduce it here
https://github.com/WeihanLi/SamplesInPractice/blob/main/HttpClientTest/AsyncEnumerableSample.cs#L13
hope it helps

[MihaZupan]

When you return an object from the controller (in this case an IAsyncEnumerable), it's serialized to the response body stream via JsonSerializer.SerializeAsync(body, yourObject).
SerializeAsync does not flush between writes, it'll only call WriteAsync on the output stream.

When you're not using compression, the response body stream is Kestrel's HTTP/1.1 implementation where every write does an implicit flush.

But if you add other stream implementations in the middle, they might buffer some data unless you call flush. Compression streams are like that, where you must manually flush them if you want "live" data.

You can change your example to call DisableBuffering after inserting the response compression middleware, which will force it to start flushing after every write.

app.UseResponseCompression();
app.Use((context, next) =>
{
    context.Features.Get<IHttpResponseBodyFeature>()?.DisableBuffering();
    return next();
});

[WeihanLi]

@MihaZupan working with it, thanks very much

[WeihanLi]

When the HttpClient with AutomaticDecompression option, it would send requests with the header: Accept-Encoding: gzip, deflate, br, it would cause the response be compressed since the client supports decompression explicitly that the server support, then as @MihaZupan mentioned above, the response would not be flushed in time, when DisableBuffering it would flushed after each write

https://github.com/dotnet/aspnetcore/blob/main/src/Middleware/ResponseCompression/src/ResponseCompressionBody.cs#L180