-
Notifications
You must be signed in to change notification settings - Fork 870
Producer
The most important error checking you should be doing is checking the result of each produce call. In the case of ProduceAsync, any error will be exposed in the form of a KafkaException if you await the call, else it will show up in the Tasks Exception property (with the IsFaulted property set to true). In the case of Produce, errors are exposed via the Error property on the DeliveryReport instance returned in the callback.
Note: when transactions become available in v1.4, you will typically be able to rely on the result of the transaction, commit - won't need to consider the produce delivery reports for errors if you are using transactions.
Events sent to the error event handler and log event handler are typically informational in nature - you can typically ignore them. In very rare cases, events on the error handler may be marked as fatal (IsFatal set to true). This can occur if idempotence or transaction semantics cannot be satisfied due to an unlikely scenario on the server (e.g. unexpected log truncation), or bug in the system.
The word 'error', and even 'fatal' are used a fair bit in the logs in benign situations. Don't be easily alarmed by this - if the producer is still operating, these are likely nothing to worry about.
A commonly adjusted configuration property on the producer is LingerMs - the minimum time between batches of messages being sent to the cluster. Larger values allow for more batching, which increases throughput. Smaller values may reduce latency. The tradeoff is complicated somewhat as throughput increases though because a smaller LingerMs will result in more broker requests, putting greater load on the server, which in turn may reduce throughput.
A good general purpose setting is 5 (the default is 0.5).
If your throughput is very low, and you really care about latency, you should probably set SocketNagleDisable to true.
Set EnableIdempotence to true. This has very little overhead - there's not much downside to having this on by default
Before the idempotent producer was available, you could achieve this by setting MaxInFlight to 1 (at the expense of reduced throughput). Messages are sent in the same order as produced. However, when idempotence is not enabled, in case of failure (temporary network failure for example), Confluent.Kafka will try to resend the data (if the Retries config property is set to > 0, the default is 2), which can cause reordering (note that this will not happen frequently, only in case of failure).