Librdkafka handles KAFKA_STORAGE_ERROR (56) as not retriable

[ccf]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

We are observing librdkafka not handle KAFKA_STORAGE_ERROR as a retriable error, as documented in the Kafka protocol. The client returns the error in the delivery callback, rather than retrying.

How to reproduce

We use the situation that caused us to discover the problem -- we exhaust file descriptors on a broker causing stress and leadership change events. Once the broker reaches the maximum number of fd's and generates KAFKA_STORAGE_ERROR to clients, they will exit accordingly.

We have reproduced this with all versions including 1.5.0.

We have simulated the same failure scenario with Java clients and the Java client handles the error as retriable.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 1.5.0
• Apache Kafka version: 2.4.2
• librdkafka client configuration: defaults
• Operating system: RHEL 7.6
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue: If this means the client can't handle a disk failure gracefully, I would say so...