Error handling

Error handling and propagation

Typically librdkafka hides any non-permanent errors from the application and handles them gracefully internally by retrying.

Permanent errors are handled in different ways:

• global errors, e.g., all brokers down, are passed to the error_cb callback, you need to call rd_kafka_poll() regularly to receive them (e.g. RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN).
• Producer: message delivery failures (and success) are passed to the dr_cb/dr_msg_cb callback, you need to call rd_kafka_poll() regularly to receive them
• Consumer: Consumer errors, such as mild notifications (RD_KAFKA_RESP_ERR__PARTITION_EOF) or more serious errors are passed as rkmessage->err for the .._consume*() functions that return or pass an rd_kafka_message_t.

This means that a producer application should be calling rd_kafka_poll() every now and then, and a consumer application should both consume messages and call rd_kafka_poll() every now and then.

If using the new high-level KafkaConsumer an application should call rd_kafka_poll_set_consumer() after creating the CONSUMER instance to automatically redirect all events to the single rd_kafka_consumer_poll() queue endpoint. You still need to use an error callback in this case to get global errors.