ERROR|rdkafka#producer-1| request(s) timed out: disconnect (average rtt

[ranjitkumar2k10]

Description

I have c++ process which acts as producer for 1 topic and consumer for some other topic .

I have 1 kafka broker and i increased the kafka partitions to 3 still i am facing this problem .

some time c++ librdkafka producer is giving errors like this.
%3|1517317005.596|FAIL|rdkafka#producer-1| [thrd:192.168.122.189:9092/1]: 192.168.122.189:9092/1: 783 request(s) timed out: disconnect (average rtt 17153.227ms)
%3|1517317005.596|ERROR|rdkafka#producer-1| [thrd:192.168.122.189:9092/1]: 192.168.122.189:9092/1: 783 request(s) timed out: disconnect (average rtt 17153.227ms)
%3|1517317007.532|FAIL|rdkafka#producer-1| [thrd:192.168.122.189:9092/1]: 192.168.122.189:9092/1: 1834 request(s) timed out: disconnect (average rtt 17137.621ms)
%3|1517317007.532|ERROR|rdkafka#producer-1| [thrd:192.168.122.189:9092/1]: 192.168.122.189:9092/1: 1834 request(s) timed out: disconnect (average rtt 17137.621ms)
%3|1517317010.564|FAIL|rdkafka#producer-1| [thrd:192.168.122.189:9092/1]: 192.168.122.189:9092/1: 15 request(s) timed out: disconnect (average rtt 17119.293ms)
%3|1517317010.565|ERROR|rdkafka#producer-1| [thrd:192.168.122.189:9092/1]: 192.168.122.189:9092/1: 15 request(s) timed out: disconnect (average rtt 17119.293ms)

How to reproduce

1. create test c++ program with kafka topic with sending 200 bytes of data and send 1000 messages/sec after some time this message will come.

NOTE: i have not created any test problem, i have already in build code which is doing the same.

Checklist

librdkafka version v0.11.0
Apache Kafka version: v2.11-0.11.0.2

[edenhill]

You are getting some really bad rtt to the broker which could be caused poor batching on the producer:
Try setting linger.ms=100 on the producer config and see if it helps

[ranjitkumar2k10]

Hi edenhil,
Thanks for your replay.. i set this linger.ms=100 parameter and now i am not getting this error.

But i am seeing 2 problems now.
This particular cpp process is having one producer for 1 topic and 1 consumer for different topic .

Problem -1:
some times i am seeing below error in producer side.
%3|1517461528.760|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Receive failed: Disconnected
%3|1517461528.760|ERROR|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Receive failed: Disconnected

Problems-2:
The same process is crashing in consumer stack as below
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./bin/C2XGeoServer -c config/server.xml -f config/log4cpp.properties'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ffce2cd6090 in ?? ()
(gdb) bt
#0 0x00007ffce2cd6090 in ?? ()
#1 0x00007f664d34380f in RdKafka::error_cb_trampoline (rk=, err=,
reason=0x7f663c0010f0 "localhost:9092/bootstrap: Receive failed: Disconnected", opaque=0x26ba048) at HandleImpl.cpp:78
#2 0x00007f664d06b68c in rd_kafka_poll_cb (rk=0x26c3680, rkq=0x26c44b0, rko=0x7f663c000ff0, cb_type=1277701169, cb_type@entry=1006637040, opaque=0x4, opaque@entry=0x0)
at rdkafka.c:2592
#3 0x00007f664d06bc57 in rd_kafka_consume0 (rk=0x26c3680, rkq=0x26c44b0, timeout_ms=) at rdkafka.c:1958
#4 0x00007f664d3467ba in RdKafka::KafkaConsumerImpl::consume (this=, timeout_ms=) at KafkaConsumerImpl.cpp:112
#5 0x000000000050567d in geoip::KafkaQueue::consumeLoop() ()
#6 0x0000000000516099 in boost::detail::thread_data<void (*)()>::run() ()
#7 0x00007f664dfe899a in thread_proxy () from /usr/local/lib/libboost_thread.so.1.58.0
#8 0x00007f664d862064 in start_thread (arg=0x7f6645fb7700) at pthread_create.c:309
#9 0x00007f664b9e862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

I have 1 kafka broker in my system, can you please suggest any broker tuning parameters and any suggestible minimum HW configuration like cores and RAM for handing 10000 messages and each message size of 200 bytes.

[edenhill]

1. https://github.com/edenhill/librdkafka/wiki/FAQ#why-am-i-seeing-receive-failed-disconnected

2. Your EventCb class instance might have gone out of scope, did you allocate it on the stack?