Skip to content

produce loss data #851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 of 7 tasks
caseware66 opened this issue May 1, 2020 · 1 comment
Closed
2 of 7 tasks

produce loss data #851

caseware66 opened this issue May 1, 2020 · 1 comment

Comments

@caseware66
Copy link

caseware66 commented May 1, 2020 •
edited
Loading

Description

When we try to write 10 messages in a Kafka cluster, we do not find the expected account as a result. We have 1 OR 2 OR 3...messages.
Do you have an idea ?

How to reproduce

import confluent_kafka
import time
import sys

producer_timings = {}
consumer_timings = {}

def error_cb(kafka_error):
print("ERR", kafka_error.name(), kafka_error.str(), file=sys.stderr)

def delivery_cb(err, msg):
if err:
print('message failed:', err, file=sys.stderr)

def calculate_thoughput(timing, n_messages=1000000, msg_size=100):
print("Processed {0} messsages in {1:.2f} seconds".format(n_messages, timing))
print("{0:.2f} MB/s".format((msg_size * n_messages) / timing / (1024*1024)))
print("{0:.2f} Msgs/s".format(n_messages / timing))

def confluent_kafka_producer_performance():
msg_count = 100
msg_size = 100
msg_payload = ('kafkatest' * 20).encode()[:msg_size]
topic = "short_name_new"

prod_config = {
    "error_cb": error_cb,
    "on_delivery": delivery_cb,
    "bootstrap.servers": "10.20.7.12:9092,10.20.7.14:9092,10.20.7.18:9092",
    "group.id": "python_inector",
    "retry.backoff.ms": 3000,
    "retries": 5,
    "default.topic.config": {"request.required.acks": "all"},
    "max.in.flight.requests.per.connection": 1,
    "queue.buffering.max.messages": 100000,
    "batch.num.messages": 100,
    "message.max.bytes": 2000000,
    "debug":"broker,topic,metadata"
}

producer = confluent_kafka.Producer(**prod_config)

producer_start = time.time()

for i in range(msg_count):
    while True:
        try:
            producer.produce(topic, value=msg_payload)
            producer.poll(0)
            break
        except BufferError as e:
            print(e, file=sys.stderr)
            producer.poll(1)

producer.flush()

return time.time() - producer_start

producer_timings["confluent_kafka_producer"] = confluent_kafka_producer_performance()
calculate_thoughput(producer_timings["confluent_kafka_producer"])

Checklist

Please provide the following information:

  • confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
    confluent-kafka-python : ('1.4.1', 17039616)
    librdkafka version:('1.4.0', 17039615)

  • Apache Kafka broker version:
    confluent_kafka 5.2.2

  • Client configuration: {...}

  • Operating system:

  • Provide client logs (with 'debug': '..' as necessary)
    %7|1588299512.083|BROKER|rdkafka#producer-1| [thrd:app]: 10.20.7.12:9092/bootstrap: Added new broker with NodeId -1
    %7|1588299512.083|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
    %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:app]: 10.20.7.12:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
    %7|1588299512.083|BRKMAIN|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Enter main broker thread
    %7|1588299512.083|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.4.0 (0x10400ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, GCC GXX PKGCONFIG INSTALL GNULD LDS LIBDL PLUGINS ZLIB SSL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0xe)
    %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Received CONNECT op
    %7|1588299512.083|STATE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
    %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: broker in state TRY_CONNECT connecting
    %7|1588299512.083|STATE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
    %7|1588299512.083|TOPIC|rdkafka#producer-1| [thrd:app]: New local topic: short_name_new
    %7|1588299512.083|TOPPARNEW|rdkafka#producer-1| [thrd:app]: NEW short_name_new [-1] 0xaa8ce0 (at rd_kafka_topic_new0:438)
    %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Connecting to ipv4#10.20.7.12:9092 (plaintext) with socket 7
    %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Connected to ipv4#10.20.7.12:9092
    %7|1588299512.083|CONNECTED|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Connected (kafkatest, refactoring, and bug fixes #1)
    %7|1588299512.083|FEATURE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
    %7|1588299512.083|STATE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
    %7|1588299512.083|PROTOERR|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1942) (incorrect broker.version.fallback?)
    %7|1588299512.083|PROTOERR|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: ApiArrayCnt -1 out of range
    %7|1588299512.083|APIVERSION|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
    %7|1588299512.084|FEATURE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
    %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: Hinted cache of 1/1 topic(s) being queried
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: Requesting metadata for 1/1 topics: connected
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Request metadata for 1 topic(s): connected
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ===== Received metadata (for 1 requested topics): connected =====
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: 3 brokers, 1 topics
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299512.084|BROKER|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Added new broker with NodeId 0
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299512.084|BRKMAIN|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Enter main broker thread
    %7|1588299512.084|BROKER|rdkafka#producer-1| [thrd:main]: kafka-3:9092/2: Added new broker with NodeId 2
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299512.084|BRKMAIN|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Enter main broker thread
    %7|1588299512.084|BROKER|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Added new broker with NodeId 1
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Topic #0/1: short_name_new with 3 partitions
    %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:main]: Topic short_name_new changed state unknown -> exists
    %7|1588299512.084|PARTCNT|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition count changed from 0 to 3
    %7|1588299512.084|TOPPARNEW|rdkafka#producer-1| [thrd:main]: NEW short_name_new [0] 0x7fe9400059c0 (at rd_kafka_topic_partition_cnt_update:747)
    %7|1588299512.084|TOPPARNEW|rdkafka#producer-1| [thrd:main]: NEW short_name_new [1] 0x7fe940005f20 (at rd_kafka_topic_partition_cnt_update:747)
    %7|1588299512.084|TOPPARNEW|rdkafka#producer-1| [thrd:main]: NEW short_name_new [2] 0x7fe940006480 (at rd_kafka_topic_partition_cnt_update:747)
    %7|1588299512.084|BRKMAIN|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Enter main broker thread
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [0]: delegate to broker kafka-3:9092/2 (rktp 0x7fe9400059c0, term 0, ref 2)
    %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [0]: delegating to broker kafka-3:9092/2 for partition with 0 messages (0 bytes) queued
    %7|1588299512.084|BRKMIGR|rdkafka#producer-1| [thrd:main]: Migrating topic short_name_new [0] 0x7fe9400059c0 from (none) to kafka-3:9092/2 (sending PARTITION_JOIN to kafka-3:9092/2)
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [1]: delegate to broker kafka-1:9092/0 (rktp 0x7fe940005f20, term 0, ref 2)
    %7|1588299512.084|TOPBRK|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Topic short_name_new [0]: joining broker (rktp 0x7fe9400059c0, 0 message(s) queued)
    %7|1588299512.084|FETCHADD|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Added short_name_new [0] to active list (1 entries, opv 0, 0 messages queued): joining
    %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [1]: delegating to broker kafka-1:9092/0 for partition with 0 messages (0 bytes) queued
    %7|1588299512.084|BRKMIGR|rdkafka#producer-1| [thrd:main]: Migrating topic short_name_new [1] 0x7fe940005f20 from (none) to kafka-1:9092/0 (sending PARTITION_JOIN to kafka-1:9092/0)
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [2]: delegate to broker kafka-2:9092/1 (rktp 0x7fe940006480, term 0, ref 2)
    %7|1588299512.084|TOPBRK|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Topic short_name_new [1]: joining broker (rktp 0x7fe940005f20, 0 message(s) queued)
    %7|1588299512.084|FETCHADD|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Added short_name_new [1] to active list (1 entries, opv 0, 0 messages queued): joining
    %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [2]: delegating to broker kafka-2:9092/1 for partition with 0 messages (0 bytes) queued
    %7|1588299512.084|BRKMIGR|rdkafka#producer-1| [thrd:main]: Migrating topic short_name_new [2] 0x7fe940006480 from (none) to kafka-2:9092/1 (sending PARTITION_JOIN to kafka-2:9092/1)
    %7|1588299512.084|PARTCNT|rdkafka#producer-1| [thrd:main]: Partitioning 100 unassigned messages in topic short_name_new to 3 partitions
    %7|1588299512.084|TOPBRK|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Topic short_name_new [2]: joining broker (rktp 0x7fe940006480, 0 message(s) queued)
    %7|1588299512.084|FETCHADD|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Added short_name_new [2] to active list (1 entries, opv 0, 0 messages queued): joining
    %7|1588299512.084|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 1 message(s) queued but broker not up
    %7|1588299512.084|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 3 message(s) queued but broker not up
    %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Broker changed state INIT -> TRY_CONNECT
    %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Broker changed state INIT -> TRY_CONNECT
    %7|1588299512.084|CONNECT|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: broker in state TRY_CONNECT connecting
    %7|1588299512.084|UAS|rdkafka#producer-1| [thrd:main]: 100/100 messages were partitioned in topic short_name_new
    %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Broker changed state TRY_CONNECT -> CONNECT
    %7|1588299512.084|CONNECT|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: broker in state TRY_CONNECT connecting
    %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Broker changed state TRY_CONNECT -> CONNECT
    %7|1588299512.084|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 18 message(s) queued but broker not up
    %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Broker changed state INIT -> TRY_CONNECT
    %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: broker in state TRY_CONNECT connecting
    %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Broker changed state TRY_CONNECT -> CONNECT
    %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: 1/1 requested topic(s) seen in metadata
    %7|1588299512.085|CLUSTERID|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ClusterId update "" -> "Prq_BeAAT4693Lia4LGWVw"
    %7|1588299512.085|CONTROLLERID|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ControllerId update -1 -> 0
    %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Connecting to ipv4#10.20.7.18:9092 (plaintext) with socket 14
    %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Connecting to ipv4#10.20.7.12:9092 (plaintext) with socket 15
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 36 message(s) queued but broker not up
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 36 message(s) queued but broker not up
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 30 message(s) queued but broker not up
    %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Connecting to ipv4#10.20.7.14:9092 (plaintext) with socket 16
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 30 message(s) queued but broker not up
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 34 message(s) queued but broker not up
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 34 message(s) queued but broker not up
    %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Connected to ipv4#10.20.7.12:9092
    %7|1588299512.085|CONNECTED|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Connected (kafkatest, refactoring, and bug fixes #1)
    %7|1588299512.085|FEATURE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Updated enabled protocol features +ApiVersion to ApiVersion
    %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Broker changed state CONNECT -> APIVERSION_QUERY
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 30 message(s) queued but broker not up
    %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Connected to ipv4#10.20.7.18:9092
    %7|1588299512.085|CONNECTED|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Connected (kafkatest, refactoring, and bug fixes #1)
    %7|1588299512.085|FEATURE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Updated enabled protocol features +ApiVersion to ApiVersion
    %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Broker changed state CONNECT -> APIVERSION_QUERY
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 36 message(s) queued but broker not up
    %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Connected to ipv4#10.20.7.14:9092
    %7|1588299512.085|CONNECTED|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Connected (kafkatest, refactoring, and bug fixes #1)
    %7|1588299512.085|FEATURE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Updated enabled protocol features +ApiVersion to ApiVersion
    %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Broker changed state CONNECT -> APIVERSION_QUERY
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 34 message(s) queued but broker not up
    %7|1588299512.085|PROTOERR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1942) (incorrect broker.version.fallback?)
    %7|1588299512.085|PROTOERR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: ApiArrayCnt -1 out of range
    %7|1588299512.085|APIVERSION|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
    %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 30 message(s) queued but broker not up
    %7|1588299512.085|FEATURE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
    %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Broker changed state APIVERSION_QUERY -> UP
    %7|1588299512.085|METADATA|rdkafka#producer-1| [thrd:kafka-1:9092/0]: Skipping metadata refresh of 1 topic(s): connected: already being requested
    %7|1588299512.086|PROTOERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1942) (incorrect broker.version.fallback?)
    %7|1588299512.086|PROTOERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ApiArrayCnt -1 out of range
    %7|1588299512.086|APIVERSION|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
    %7|1588299512.086|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 36 message(s) queued but broker not up
    %7|1588299512.086|FEATURE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
    %7|1588299512.086|STATE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Broker changed state APIVERSION_QUERY -> UP
    %7|1588299512.086|METADATA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Skipping metadata refresh of 1 topic(s): connected: already being requested
    %7|1588299512.086|PROTOERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1942) (incorrect broker.version.fallback?)
    %7|1588299512.086|PROTOERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ApiArrayCnt -1 out of range
    %7|1588299512.086|APIVERSION|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
    %7|1588299512.086|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 34 message(s) queued but broker not up
    %7|1588299512.086|FEATURE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
    %7|1588299512.086|STATE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Broker changed state APIVERSION_QUERY -> UP
    %7|1588299512.086|METADATA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: Skipping metadata refresh of 1 topic(s): connected: already being requested
    %7|1588299512.086|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299512.086|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299512.086|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
    %7|1588299512.086|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299512.086|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Request metadata for 1 topic(s): partition leader query
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ===== Received metadata (for 1 requested topics): partition leader query =====
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 3 brokers, 1 topics
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Topic #0/1: short_name_new with 3 partitions
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 1/1 requested topic(s) seen in metadata
    %7|1588299515.087|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299515.087|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299515.087|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
    %7|1588299515.088|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299515.088|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Request metadata for 1 topic(s): partition leader query
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ===== Received metadata (for 1 requested topics): partition leader query =====
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: 3 brokers, 1 topics
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Topic #0/1: short_name_new with 3 partitions
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: 1/1 requested topic(s) seen in metadata
    %7|1588299518.089|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299518.089|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299518.089|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
    %7|1588299518.089|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299518.089|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299519.082|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
    %7|1588299519.082|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Request metadata for 1 topic(s): partition leader query
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ===== Received metadata (for 1 requested topics): partition leader query =====
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 3 brokers, 1 topics
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Topic #0/1: short_name_new with 3 partitions
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 1/1 requested topic(s) seen in metadata
    %7|1588299521.090|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299521.090|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299521.090|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
    %7|1588299521.091|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299521.091|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299522.082|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
    %7|1588299522.082|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Request metadata for 1 topic(s): partition leader query
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ===== Received metadata (for 1 requested topics): partition leader query =====
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 3 brokers, 1 topics
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Topic #0/1: short_name_new with 3 partitions
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 1/1 requested topic(s) seen in metadata
    %7|1588299524.092|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299524.092|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299524.092|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
    %7|1588299524.093|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299524.093|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299525.083|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
    %7|1588299525.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Request metadata for 1 topic(s): partition leader query
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ===== Received metadata (for 1 requested topics): partition leader query =====
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 3 brokers, 1 topics
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Topic #0/1: short_name_new with 3 partitions
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 1/1 requested topic(s) seen in metadata
    %7|1588299527.093|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299527.093|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299527.093|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
    %7|1588299527.094|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299527.094|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Request metadata for 1 topic(s): partition leader query
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ===== Received metadata (for 1 requested topics): partition leader query =====
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 3 brokers, 1 topics
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Topic #0/1: short_name_new with 3 partitions
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 1/1 requested topic(s) seen in metadata
    %7|1588299530.095|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299530.095|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299530.095|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
    %7|1588299530.097|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299530.097|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Request metadata for 1 topic(s): partition leader query
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ===== Received metadata (for 1 requested topics): partition leader query =====
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 3 brokers, 1 topics
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Topic #0/1: short_name_new with 3 partitions
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 1/1 requested topic(s) seen in metadata
    %7|1588299533.096|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299533.096|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299533.096|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
    %7|1588299533.099|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
    %7|1588299533.099|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
    %7|1588299534.083|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
    %7|1588299534.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Request metadata for 1 topic(s): partition leader query
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ===== Received metadata (for 1 requested topics): partition leader query =====
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 3 brokers, 1 topics
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker #0/3: kafka-1:9092 NodeId 0
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Topic #0/1: short_name_new with 3 partitions
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
    %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 1/1 requested topic(s) seen in metadata

  • Provide broker log excerpts

  • Critical issue
@edenhill
Copy link
Contributor

edenhill commented May 1, 2020

Your cluster state is messed up, the partition leader for partition 0 as reported in Metadata is broker 2, but broker 2 responds to Produce requests with the error saying it is not the leader.
Suggest you look into the health of the Kafka cluster.

@jliunyu jliunyu mentioned this issue Mar 4, 2022
Open
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@edenhill @caseware66