Receive failed from one broker causes coredump

[jimmy1911]

I use consumer in my server, program like this:

/*****************************************************/
RdKafka::Topic *topic = RdKafka::Topic::create(consumer, consume_req.topic,tconf, errstr);
RdKafka::ErrorCode resp = consumer->start(topic, consume_req.partition_id, consume_req.offset);
RdKafka::Message *msg = consumer->consume(topic, consume_req.partition_id, 1000);
msg_consume(msg, NULL,&_return);
consumer->poll(0);
consumer->stop(topic, consume_req.partition_id);
consumer->poll(1000);
/*****************************************************/
But after running minutes, core dump occured. Backtrace info is like this:

(gdb) bt
#0  0x0000000000544d59 in vtable for boost::detail::sp_counted_impl_p<apache::thrift::server::TNonblockingIOThread> ()
#1  0x000000000052492f in RdKafka::log_cb_trampoline (rk=<value optimized out>, level=3, fac=<value optimized out>, 
    buf=0x7ff3a4944560 "10.10.38.70:9503/3: Receive failed: Disconnected") at HandleImpl.cpp:55
#2  0x00000000005254a6 in rd_kafka_log0 (rk=0x7ce8d0, extra=<value optimized out>, level=3, fac=0x55671c "FAIL", 
    fmt=<value optimized out>) at rdkafka.c:141
#3  0x000000000052fe9d in rd_kafka_broker_fail (rkb=0x7cf880, err=RD_KAFKA_RESP_ERR__TRANSPORT, fmt=0x55679a "Receive failed: %s")
    at rdkafka_broker.c:419
#4  0x0000000000530598 in rd_kafka_recv (rkb=0x7cf880) at rdkafka_broker.c:1540
#5  0x00000000005313c0 in rd_kafka_broker_io_serve (rkb=0x7cf880) at rdkafka_broker.c:2527
#6  0x0000000000533319 in rd_kafka_broker_consumer_serve (arg=0x7cf880) at rdkafka_broker.c:4185
#7  rd_kafka_broker_thread_main (arg=0x7cf880) at rdkafka_broker.c:4236
#8  0x00007ff3a72b19d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ff3a623c9dd in clone () from /lib64/libc.so.6

This seems like that Receive failed from one broker causes coredump. If so, how to avoid this?

[edenhill]

It seems like it crashes in boost or thrift, you need to find the cause of the crash, e.g., what invalid memory is being accessed?

[jimmy1911]

Thanks for your suggestion. Another question, I need to use both consumer and producer in my kafka proxy.Can consumer and producer share the same conf and tconf instance.

[edenhill]

Yes, config objects can be reused in the C++ interface.
In the C interface they need to be duplicated/copied with .._conf_dup().

[jimmy1911]

Thank you very much.