confusing client behavior after raise "Topic authorization failed" error

[icearn]

Description

Test on exceptional cases, such as produce to/consume from a non-existing topic, or an existing topic with no ACL permessions. Under the debug mode, I've received the error "Broker: Topic authorization failed", however my client continues after this and try query the kafka cluster endlessly.

Wasn't it supposed to throw an exception and stop? My understanding is, this type of error is fatal for the client and before the circumstance changed there is no meaning to retry querying the kafka cluster.

How to reproduce

Setup a non-existing topic name for your client and start to run it

Checked I am on current latest release 0.11.6

Checklist

Please provide the following information:

• librdkafka version (release number or git tag): <0.11.6>
• Apache Kafka version: <2.0>
• librdkafka client configuration: "client.id": "TestConsumer1", "bootstrap.servers": "kafka1:9093,kafka2:9093,kafka3:9093", "debug": "all", //"group.id": "enable.auto.offset.store": false, //TO enable large volume consuming - not suite for exactly once delievery "message.timeout.ms": 10000, "auto.commit.interval.ms": 2000, "auto.offset.reset": "earliest", "ssl.ca.location": "Trusted-Certs.pem", "security.protocol": "Sasl_Ssl"
• Operating system: <Centos 5 (x64)>
• Provide logs (with debug=.. as necessary) from librdkafka

2019-02-05 09:23:08.981 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Hinted cache of 1/1 topic(s) being queried
2019-02-05 09:23:08.981 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: consumer join: metadata for subscription only available for 0/1 topics (-1ms old)
2019-02-05 09:23:08.981 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Request metadata for 1 topic(s): consumer join
2019-02-05 09:23:08.982 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": postponing join until up-to-date metadata is available
2019-02-05 09:23:08.982 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": no topics in metadata matched subscription
2019-02-05 09:23:08.982 [Debug] From(TestConsumer1#consumer-1) [thrd:sasl_ssl://kafka3:9093/bootstrap]: sasl_ssl://kafka3:9093/3: Sent MetadataRequest (v2, 49 bytes @ 0, CorrId 27)
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:sasl_ssl://kafka3:9093/bootstrap]: sasl_ssl://kafka3:9093/3: Received MetadataResponse (v2, 261 bytes, CorrId 27, rtt 1.59ms)
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: ===== Received metadata (for 1 requested topics): consumer join =====
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: ClusterId: B-ZyAAnZQkGxGJW_yGZecw, ControllerId: 3
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: 4 brokers, 1 topics
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Broker #0/4: kafka2:9093 NodeId 2
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Broker #1/4: kafka4:9093 NodeId 4
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Broker #2/4: kafka1:9093 NodeId 1
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Broker #3/4: kafka3:9093 NodeId 3
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Topic #0/1: TheEXistingTopic with 0 partitions: Broker: Topic authorization failed
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: 1/1 requested topic(s) seen in metadata
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Purged 1/1 cached topic hint(s)
2019-02-05 09:23:08.984 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": no topics in metadata matched subscription
2019-02-05 09:23:08.985 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": no topics in metadata matched subscription
2019-02-05 09:23:10.247 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: OffsetCommit internal error: Local: No offset stored
2019-02-05 09:23:10.247 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored
2019-02-05 09:23:10.247 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": unassign done in state up (join state init): without new assignment: OffsetCommit done (__NO_OFFSET)
2019-02-05 09:23:11.437 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": join with 0 (1) subscribed topic(s)
2019-02-05 09:23:11.437 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Hinted cache of 1/1 topic(s) being queried
2019-02-05 09:23:11.437 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: consumer join: metadata for subscription only available for 0/1 topics (-1ms old)
2019-02-05 09:23:11.437 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Request metadata for 1 topic(s): consumer join
2019-02-05 09:23:11.438 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": postponing join until up-to-date metadata is available
2019-02-05 09:23:11.438 [Debug] From(TestConsumer1#consumer-1) [thrd:sasl_ssl://kafka3:9093/bootstrap]: sasl_ssl://kafka3:9093/3: Sent MetadataRequest (v2, 49 bytes @ 0, CorrId 28)
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:sasl_ssl://kafka3:9093/bootstrap]: sasl_ssl://kafka3:9093/3: Received MetadataResponse (v2, 261 bytes, CorrId 28, rtt 3.65ms)
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: ===== Received metadata (for 1 requested topics): consumer join =====
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: ClusterId: B-ZyAAnZQkGxGJW_yGZecw, ControllerId: 3
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: 4 brokers, 1 topics
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Broker #0/4: kafka2:9093 NodeId 2
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Broker #1/4: kafka4:9093 NodeId 4
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Broker #2/4: kafka1:9093 NodeId 1
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Broker #3/4: kafka3:9093 NodeId 3
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: Topic #0/1: TheEXistingTopic with 0 partitions: Broker: Topic authorization failed
2019-02-05 09:23:11.442 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: sasl_ssl://kafka3:9093/3: 1/1 requested topic(s) seen in metadata
2019-02-05 09:23:11.443 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Purged 1/1 cached topic hint(s)
2019-02-05 09:23:11.443 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": no topics in metadata matched subscription
2019-02-05 09:23:11.443 [Debug] From(TestConsumer1#consumer-1) [thrd:main]: Group "ConsumerSampleGroup": no topics in metadata matched subscription

• Provide broker log excerpts
• Critical issue

[edenhill]

librdkafka treats all errors as temporary, which will eventually be resolved. This approach provides robustness when upgrading, reconfiguring, etc, the cluster, since a config mistake on one broker will not take down the entire client fleet.

But, if an application wants to react on these type of errors it may do so by registering an error_cb callback to catch general/global errors, or check the message error on the consumer, or the delivery report error on the producer.

I believe the only case where there is no error propagated is when the consumer is subscribing to a non-existent topic (there is an existing issue for that), and possibly also when topic auth fails as seen here?

[edenhill]

Consumer behaviour is tracked in #1540