Stream Batching

[kiro]

Hi, we noticed 6x throughput improvement if we stream 100 byte messages in a batch of 50 (e.g. one response message with repeated field) rather than streaming them one by one. So are there plans to add support for batching in the streaming implementation?

Thanks,
Kiril

[iamqizhao]

This will be tricky because batching might enlarge latency. We have some minimal deliberate batching support in the code already. But how to enhance it to maximize throughput without hurting latency is complex and need more time. Performance would be one of our main focus next quarter.

[xiang90]

@iamqizhao Are we open to add a Flush() method to stream to enable user controlled batching?

[iamqizhao]

This would be tricky on the interaction with Flush decision on the transport layer.

[bradfitz]

@iamqizhao, Go's http2 server does this aggregation automatically. It has a write scheduler that only flushes only when a packet is full or there's nothing else to send. So if we switch, I think this should get a lot easier to control. (or ideally: not control, if it just works)

[iamqizhao]

@bradfitz , as I mentioned, the current grpc transport does this automatically too (may be different from what http2 does). The problem is that whether Flush knob should be exposed to users.

[sailat]

We are working on batching requests this quarter to improve perf. Will update this thread with available date soon.

[hsaliak]

@dfawley assigning to you to triage and make a decision on exposing the Flush knob to users.

[dfawley]

We discussed this today at some length. Without major API changes or hacks (i.e. passing a magic wrapper type to stream.SendMsg), there isn't much we can do here. It does sound like a good idea, though, so I'll leave it open.

[MakMukhi]

We changed out write mechanism to use a dedicated loop to write which essentially solves the batching problem.