Delay when calling Producer.flush the first time after calling Producer.produce

[tribeiro]

Description

This issue is the same as #1581. With confluent_kafka >2 a simple script to produce kafka message experiences a 1s delay when calling Producer.flush the first time after a Producer.produce.

How to reproduce

Run the attached code , after setting KAFKA_BROKER_ADD appropriately.

minimal_write.py.tgz

You can unpack it with tar zxvf minimal_write.py.tgz.

Then run python minimal_write.py.

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):

  confluent_kafka.version(): ('2.5.0', 33882112)
  confluent_kafka.libversion(): ('2.5.0', 33882367)

  Installed with conda.

• Apache Kafka broker version:

  Running kafka with Strimzi docker image: quay.io/strimzi/kafka:0.36.1-kafka-3.5.1

  Docker compose configuration for the kafka cluster: docker-compose.yaml.tgz

• Client configuration: {...}

  There is no client but the producer configuration is

     {
         "acks": acks,
         "queue.buffering.max.ms": 0,
         "bootstrap.servers": KAFKA_BROKER_ADDR,
         "api.version.request": True,
     }
  

• Operating system:

  Tested on Mac OS and Linux Alma8.

• Provide client logs (with 'debug': '..' as necessary)

• Provide broker log excerpts

• Critical issue

[tribeiro]

Ok, after investigating this in depth I finally managed to find the solution. When we call producer.list_topics() we should provide a list of the topics name we want to list. This makes sure the producer updates the metadata for the appropriate topic and later, when we call flush for the first time, there is no delay.

I got to this after investigating some of the release notes that mentioned the 1s delay when calling flush if there is no topic metadata and then checking the list_topics method behavior. I leave this note here for future reference.