Performance: streaming performance of gRPC.AspNetCore < gRPC.Core << gRPC C++

[ccifra]

What version of gRPC and what language are you using?

gRPC C++ and gRPC for .NET (both asp.net core and legacy gRPC). Using latest stable version of nugets.
gRPC.ASPNetCore 2.33.1
gRPC 2.38.1

What operating system (Linux, Windows,...) and version?

Tested primarily on Linux but similar results on Windows

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.NET 5

What did you do?

Now that gRPC.Core is in maintenance mode we are migrating our gRPC code to use gRPC.AspNetCore. For our application the latency and throughput of simple RPC methods and streaming large data sets is very important. This is especially true on localhost connections where we use gRPC to communicate between services.

I have a performance test suite here:
https://github.com/ccifra/grpc-perf

This test suite uses a C++ client to talk to a server written with asp.net core gRPC, gPRC.Core (now in maitenance), and gRPC for C++. Here are some performance numbers that I found when using a Linux computer with a localhost connection:

ASP.NET Core gRPC
RPC Call: 22907 messages/s
Server Streaming: 940 MB/s

.NET gRPC Core (Now in maitenance)
RPC Call: 23680 messages/s
Sever Streaming: 1,242.88 MB/s

C++ gRPC
RPC Call: 49878.1 messages/s
100000 Samples: 3,603.60 MB/s

RPC Call performance between the .net implementations is roughly the same. Both are much >50% slower than C++. The streaming performance of ASP.NetCore is about 25% slower than the gRPCCore implementation. And again both are much slower than C++.

The streaming test basically writes 100,000 doubles to the client using a server streaming API. It is as simple as I know how to make it:
https://github.com/ccifra/grpc-perf/blob/7c01e01b8a2bf39464bf08c1bd788e9a8c529618/dotnet/gRPCPerfTest/PerfTestServer/PerftestServiceImpl.cs#L41

What did you expect to see?

I would expect the non-maintenance version gRPC for .NET to have similar performance than the previous version. I would hope to see the performance of .net gRPC to be much closer to C++.

What did you see instead?

Much lower performance of .NET gRPC.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

[JamesNK]

Average server streaming performance is a known issue with grpc-dotnet.

We have ideas about how to improve it, but it requires lower-level changes in Kestrel's HTTP/2 implementation. It won't happen in .NET 6 but hopefully we have time in .NET 7 (i.e. next year).

[TFTomSun]

... just to understand it: the security fixes for GRPC.Core will be stopped in spring next year. And the new and "better" implementation will "hopefully" get this basic performance improvements by end of next year?

[JamesNK]

The new implementation is faster in the other 3 call types. Almost as fast as C++.

With server streaming it is slightly slower.

We’re open source so PRs welcome if it is important to you.

[rafikiassumani-msft]

This issue will be fixed as soon as the work on the following issue is completed: dotnet/aspnetcore#30235

[Naganathan]

Can anyone let us know whether this issue has fixed or not?

Because still we are facing performance delay in our Asp.Net Core project when we using Net7.0 preview3 version. We have implemented subscribe/unsubscribe functionality and there is a lagging to receive a data for the subscribed clients (we are using 2 clients) from the server.

1. one client is frontend i.e. Electron
2. another one client is separate project in C#.

[davidfowl]

@Naganathan Did you do profiling work to figure out where the bottle neck is that you concluded that it was this problem?

[saiprabhur]

Performance issue of GRPC server streaming in the thread above has been fixed in the .net 7.0 ?

[Naganathan]

@davidfowl, responseStream.WriteAsync() method will take time to send data to each clients. That thread will wait up to writing the data to one client. So if we send like 20 data/sec for 2 or more clients means that will take much time to complete the writing.

[davidfowl]

@saiprabhur

Performance issue of GRPC server streaming in the thread above has been fixed in the .net 7.0 ?

It's improved in the next preview of .NET 7. Wait for the blog post in the next week or so.

@davidfowl, responseStream.WriteAsync() method will take time to send data to each clients. That thread will wait up to writing the data to one client. So if we send like 20 data/sec for 2 or more clients means that will take much time to complete the writing.

Do you have any profiles?

[Naganathan]

@davidfowl Please find the below attached Asp.Net core sample to get lagging issue on this

PerformanceLagging_GRPC.zip

Where,

1. BackEnd.HostApp project is a server and DataAcqClient project is a client app.
2. 1st run the Server app and then just run 3 or more DataAcqClient.exe (build and get from bin folder). In anyone of console, press any key to receive data from the server. The data will receive all subscribed clients.
3. In server, I configured and sent data upto 40sec. But in clients, will receive data more than 40sec.

[davidfowl]

@Naganathan run these benchmarks on preview4 when it comes out and you can see if it makes a difference in your setup.

[davidfowl]

In server, I configured and sent data upto 40sec. But in clients, will receive data more than 40sec.

That doesn't sound like a profile. Did you capture a profile and do any performance analysis besides the timings?

[Naganathan]

@davidfowl, We have profiling the WriteAsync() method, its take 8-12ms averagely due to the external method will take time to complete writing for a client.

That could take much time, if we were sending 40message/sec for 3 or more clients means.

As per your suggestion, we will wait for next preview of .NET 7 and ensure on that.

[davidfowl]

It's out https://devblogs.microsoft.com/dotnet/asp-net-core-updates-in-dotnet-7-preview-4/

[Naganathan]

Hi Davidfowl,

We have upgraded our project to NET7.0 latest version but still facing same issue when using responseStream.WriteAsync().

when continuously sending data to same client (using single client for testing), each WriteAsync() method will take much time to complete message (here I am posting 24 message at every 125ms).

I am sending data at every 125ms with duration for 2min so client has to receive all the data within 2min but in GRPC take 18m to complete 2min data.

Note:
when I am using signalR it is working fine as expected time duration.

we are posting continuous data instead of posting bulk data. there are more lag while using GRPC's WriteAsync().

Please let us know any other solution to achieve this scenario.