batch messages can be too large in new raft client implementation

[BusyJay]

Bug Report

New raft client implementation tends to batch more messages than before and we found it exceeded the max grpc messages in internal tests.

[2021/02/25 17:42:42.658 +08:00] [ERROR] [raft_client.rs:431] ["connection aborted"] [addr=] [receiver_err="Some(RpcFailure(RpcStatus { stat
us: 8-RESOURCE_EXHAUSTED, details: Some(\"Sent message larger than max (10705556 vs. 10485760)\") }))"] [sink_error=Some(RemoteStopped)] [store_id=7]

Calculating the message size using Message::compute_size should make it correct at the cost with extra CPU usage. We may also check what fields can be missed as even max batch size (128) is reached, there was about 1K extra size per message.

[hicqu]

How about remove the size limit but only keep the batch count limit? So the total size of a batch won't be too exorbitant and we can save CPU times from compute_size.

[BusyJay]

Perhaps we should bench what size is a reasonable good size.

[BusyJay]

I believe #8926 introduce the regression, /cc @sticnarf.

Raft client doesn't check for raft messages' context, so it's highly possible a ReadIndex requests that have a lot of ranges exceed the 10MiB limit. Raft client only allows at most 128 requests in a batch, so if there the ranges' sizes exceed 80K per messages, it will trigger the error.

This also explains why the error occurs more often than the past since v5.0.0.

[sticnarf]

@BusyJay Does it happen only when there are replica reads? The additional contexts should only exist when using replica reads.

[BusyJay]

I prints the whole messages in a test case that can reproduce the issue frequently. It turns out it's because there are too many entries batch into one messages. A batch with 30 messages can contains about 45199 entries. Each entry has about 11 extra bytes overhead, which means there are at least 497189 extra bytes, which is very close to the default extra buffer 524288.

Given now we have collected all bytes length from the pb message, now I feel confident to disable the size hard limit on the connection level to solve the issue for all. We still need a way to monitor the abnormal message size though.