issue with rdkafka_example when auto.create.topics.enable=true in kafka8

[opsmekanix]

Kafka 8 offers a feature where new topics are automatically created by the broker if a producer attempts to send messages to a non-existent topic. When 'auto.create.topics.enable=true' is set in Kafka 8, messages sent by the 'rdkafka_example producer' get black holed by the broker. FWIW: messages sent to existing topics when 'auto.create.topics.enable=true' works just fine.

To reproduce this condition, add 'auto.create.topics.enable=true' to the server.properties file and start the broker. You should find that kafka-consumer-producer.sh triggers the broker to create the new topics and works as advertised. But something goes wrong with rdkafka_example.

Thanks for continuing to maintain this project and looking into this issue.

-- Adam

[edenhill]

librdkafka currently waits for a topic to show up in the broker metadata mapping before producing a message for it.
I'll change it to attempt producing a single message to a random broker for unknown topics.

[edenhill]

This issue should now be fixed.

If the topic-object is created (with rd_kafka_topic_new()) when at least one broker is up the broker-side topic creation will be immediate, otherwise the topic will be auto-created on the broker within at most 1 second.
Produced messages will be queued and sent to the broker when the topic has been created.

Could you verify this on your end and then close the issue?

Please update your master branch, rebuild librdkafka and do a 'make clean all' in examples/, then try your test again.

Thanks

[opsmekanix]

The new patch seems to be working as I could see the new topic being created. Thanks for the quick turn around.