How to minimize the producer timeout period during partition leader failover?

[vereshchagin-d]

Hi community!

We're experiencing longer than expected producer timeouts during network partition scenarios in our Strimzi Kafka cluster. Here's our setup and test scenario:

Environment:

Cluster setup:

• Strimzi Operator version: 0.44.0
• Kafka version: 3.7.0, KRaft mode
• Cluster consists of 3 brokers and 3 controllers.
• Brokers and controllers are distributed across different nodes and 3 availability zones.

Topic configuration:

• 1 partition
• replication factor = 3
• min.insync.replicas = 2

Test scenario:

Producer Configuration:

• We used xk6-kafka as a producer.
• Configuration snippet:

    const writer = new Writer({
      brokers: brokers,
      topic: topic,
      requiredAcks: -1, // all
      balancer: BALANCER_ROUND_ROBIN,
      writeTimeout: SECOND * 2, 
      maxAttempts: 1,
      connectLogger: false
    });

• Started a producer with acks=all

Failure Simulation:

Using chaos-mesh, we simulated a network partition for the broker that was the partition leader
Observed producer errors lasting approximately 15 seconds before recovery

chaos-mesh CRD snippet:

apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  namespace: kafka-mks-test
  name: kf-mks-test
spec:
  entry: main-case
  templates:
    - name: main-case
      templateType: Serial
      children:
        - partition
    - name: partition
      templateType: NetworkChaos
      deadline: 1m
      networkChaos:
        selector:
          namespaces:
            - kafka-mks-test
          labelSelectors:
            strimzi.io/pod-name: kf-mks-test-staging-broker-v1-85923-3
        mode: one
        action: partition
        direction: to

Logs

      █  writer.produce                                                                                                                                                                                              │
│                                                                                                                                                                                                                     │
│        ✗ Exception raised "GoError: Error writing messages., OriginalError: Kafka write errors (1/1), errors: [[6] Not Leader For Partition: the client attempted to send messages to a replica that is not the lea │
│         ↳  0% — ✓ 0 / ✗ 40308                                                                                                                                                                                       │
│        ✗ Exception raised "GoError: Error writing messages., OriginalError: Kafka write errors (1/1), errors: [[7] Request Timed Out: the request exceeded the user-specified time limit in the request]"           │
│         ↳  0% — ✓ 0 / ✗ 8000                                                                                                                                                                                        │
│                                                                                                                                                                                                                     │
│      checks.............................: 0.00%       0 out of 48308                                                                                                                                                │
│      data_received......................: 0 B         0 B/s                                                                                                                                                         │
│      data_sent..........................: 0 B         0 B/s                                                                                                                                                         │
│      group_duration.....................: avg=90.84ms min=5.09ms med=92.15ms max=1.08s   p(90)=105.14ms p(95)=111.2ms                                                                                               │
│      iteration_duration.................: avg=1m32s   min=31.16s med=1m36s   max=1m46s   p(90)=1m45s    p(95)=1m45s                                                                                                 │
│      iterations.........................: 1487        6.975183/s                                                                                                                                                    │
│    ✓ kafka_reader_error_count...........: 0           0/s                                                                                                                                                           │
│      kafka_writer_acks_required.........: -1          min=-1           max=0                                                                                                                                        │
│      kafka_writer_async.................: 0.00%       0 out of 1924821                                                                                                                                              │
│      kafka_writer_attempts_max..........: 1           min=1            max=1                                                                                                                                        │
│      kafka_writer_batch_bytes...........: 3.3 GB      16 MB/s                                                                                                                                                       │
│      kafka_writer_batch_max.............: 1           min=1            max=1                                                                                                                                        │
│      kafka_writer_batch_queue_seconds...: avg=2.36ms  min=0s     med=1.15ms  max=88.41ms p(90)=6.26ms   p(95)=8.59ms                                                                                                │
│      kafka_writer_batch_seconds.........: avg=64.29µs min=3.14µs med=7.68µs  max=88.41ms p(90)=15.79µs  p(95)=26.97µs                                                                                               │
│      kafka_writer_batch_size............: 1924821     9028.903307/s                                                                                                                                                 │
│      kafka_writer_batch_timeout.........: 277h46m40s  min=277h46m40s   max=277h46m40s                                                                                                                               │
│    ✗ kafka_writer_error_count...........: 48308       226.601986/s                                                                                                                                                  │
│      kafka_writer_message_bytes.........: 3.3 GB      16 MB/s                                                                                                                                                       │
│    ✓ kafka_writer_message_count.........: 1924821     9028.903307/s                                                                                                                                                 │
│      kafka_writer_read_timeout..........: 2777h46m40s min=2777h46m40s  max=2777h46m40s                                                                                                                              │
│      kafka_writer_retries_count.........: 0           0/s                                                                                                                                                           │
│      kafka_writer_wait_seconds..........: avg=0s      min=0s     med=0s      max=0s      p(90)=0s       p(95)=0s                                                                                                    │
│      kafka_writer_write_count...........: 1924821     9028.903307/s                                                                                                                                                 │
│      kafka_writer_write_seconds.........: avg=89.92ms min=4.82ms med=91.9ms  max=1.02s   p(90)=104.71ms p(95)=110.39ms                                                                                              │
│      kafka_writer_write_timeout.........: 555h33m20s  min=555h33m20s   max=555h33m20s                                                                                                                               │
│      vus................................: 0           min=0            max=1000                                                                                                                                     │
│      vus_max............................: 1000        min=372          max=1000    

I also tried running ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic xk6_kafka_test_topic --describe on one of the brokers in a loop to see how quickly the cluster detects broker loss. Although this was a slightly different Kafka cluster (6 brokers and 3 controllers), the same issue occurred.

while true; do 
    echo "=== $(date '+%Y-%m-%d %H:%M:%S') ==="
    echo "Topic:"
    ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic xk6_kafka_test_topic --describe
    sleep 1
done
...
=== 2024-11-29 11:03:10 ===
Topic:
Topic: xk6_kafka_test_topic     TopicId: UDAxab2rQRuCHKjNN_94iw PartitionCount: 1       ReplicationFactor: 3    Configs: compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,max.message.bytes=31457300
        Topic: xk6_kafka_test_topic     Partition: 0    Leader: 3       Replicas: 3,5,2 Isr: 2,5,3
=== 2024-11-29 11:03:12 ===
Topic:
Topic: xk6_kafka_test_topic     TopicId: UDAxab2rQRuCHKjNN_94iw PartitionCount: 1       ReplicationFactor: 3    Configs: compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,max.message.bytes=31457300
        Topic: xk6_kafka_test_topic     Partition: 0    Leader: 3       Replicas: 3,5,2 Isr: 2,5,3
=== 2024-11-29 11:03:14 ===
Topic:
Topic: xk6_kafka_test_topic     TopicId: UDAxab2rQRuCHKjNN_94iw PartitionCount: 1       ReplicationFactor: 3    Configs: compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,max.message.bytes=31457300
        Topic: xk6_kafka_test_topic     Partition: 0    Leader: 3       Replicas: 3,5,2 Isr: 2,5,3
=== 2024-11-29 11:03:27 ===
Topic:
Topic: xk6_kafka_test_topic     TopicId: UDAxab2rQRuCHKjNN_94iw PartitionCount: 1       ReplicationFactor: 3    Configs: compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,max.message.bytes=31457300
        Topic: xk6_kafka_test_topic     Partition: 0    Leader: 5       Replicas: 3,5,2 Isr: 2,5
=== 2024-11-29 11:03:29 ===
Topic:
Topic: xk6_kafka_test_topic     TopicId: UDAxab2rQRuCHKjNN_94iw PartitionCount: 1       ReplicationFactor: 3    Configs: compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,max.message.bytes=31457300
        Topic: xk6_kafka_test_topic     Partition: 0    Leader: 5       Replicas: 3,5,2 Isr: 2,5

Provided logs show Kafka write errors and leader election delays, with a notable lag between 11:03:14 and 11:03:27.

Question:

We don't want to increase the producer timeout at this time, as it is a business requirement. Is there a way to reduce this recovery time? What configurations should we look into to minimize the producer timeout period during leader failover? I've noticed that strimzi operator ignores broker. settings.

We'd appreciate any insights on:

• Recommended broker/topic configurations
• Producer-side settings that might help
• Expected behavior and typical recovery times in such scenarios

Thank you in advance!

[vereshchagin-d]

@scholzj Hi! Could you please help me with some advice regarding my question? Thank you.

[scholzj]

Sorry, I do not know anything about this nor have I any idea about what xk6-kafka producer is.

[vereshchagin-d]

The question is more about Kafka fine-tuning, not xk6-kafka (it's just a tool that simulates load). For example, what if we want to tune the broker.session.timeout.ms? Do you have a workaround?
Thanks.

[scholzj]

Workaround for what? I would expect broker.session.timeout.m to be configurable in the Kafka CR. But I have no idea about what the consequences of playing with it might be.