What are the tradeoffs/downsides in reducing queue.buffering.max.ms?

[winbatch]

I found that when I made this number small as compared to the default, performance quadrupled. I realize there is no free lunch however so wanted to better understand what the downside is of doing this. Is it an increase in CPU? (Client side or server side?)

[edenhill]

Please read this section of the librdkafka introduction:
https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#high-throughput

It explains the key aspects of throughput and latency.

If your application has special send patterns you might need to configure the two two config properties "batch.num.messages" and "queue.buffering.max.ms".

[edenhill]

Did this answer your question?

[winbatch]

yes and no. Was wondering if it increases cpu on either the client machine
or broker host

On Wed, Jan 1, 2014 at 5:24 PM, Magnus Edenhill notifications@github.comwrote:

Did this answer your question?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-31431655
.

[edenhill]

Using a smaller value for "queue.buffering.max.ms" means less messages will be sent in each produce request, thus the request's overhead will be higher per-message than if more messages are sent.
It will not affect the producer side much, but the broker will only handle one produce request at the time (there are multiple requests in-flight though), so the fewer messages in a request, the lower performance.

But it also depends on your usage pattern; what rate you are producing messages at, and if they are produced in bursts.

You may need to optimize your "batch.num.messages" and "queue.buffering.max.ms" properties according to INTRODUCTION.md. That is:

• queue.buffering.max.ms - how long librdkafka will wait for batch.num.messages to be produce()d by the application before sending them to the broker in a produce request
• batch.num.messages - the maximum number of messages that will go in a single produce request to the broker.

When either one of these two properties are reached a produce request will be sent.

If you need high throughput: both these properties should be high (default),
if you need low latency: decrease queue.buffering.max.ms to your maximum accepted queue latency

[winbatch]

Thanks for the info, closing