[Consumer] rd_kafka_destory hang while upgrading librdkafka from version 0.11.4 to 2.0.2

[rpraveensriram]

We are upgrading librdkafka from 0.11.4 to 2.0.2, along with upgrading of Kafka broker from 2.8 to 3.4. As part of this upgrade, we could see many flags were deprecated and some default values are changed. But the major change, we see is broker.version.fallback changed to 0.10.0.

With the latest version of librdkafka and Kafka broker (broker.version.fallback => 0.10.0), we could see our consumer client hang and created core dump at rd_kafka_destroy. We made sure that all messages, topic, topic partitions are released before calling rd_kafka_destory. Made sure we followed the termination sequence. Adding the logs

17:24:41.622763 Mar 10 RX 93fff700 % KAFKA INTERNAL LOGGER CALLBACK : 1678433081.622 RDKAFKA-7-TERMINATE: kcea#consumer-1: [thrd:cbtest:50066/0]: cbtest:50066/0: Handle is terminating in state DOWN: 2 refcnts (0x7fcb940014f8), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
17:24:41.622768 Mar 10 RX 93fff700 % KAFKA INTERNAL LOGGER CALLBACK : 1678433081.622 RDKAFKA-7-STATE: kcea#consumer-1: [thrd:cbtest:50066/0]: cbtest:50066/0: Broker changed state DOWN -> INIT
17:24:41.622775 Mar 10 RX 93fff700 % KAFKA INTERNAL LOGGER CALLBACK : 1678433081.622 RDKAFKA-7-BROADCAST: kcea#consumer-1: [thrd:cbtest:50066/0]: Broadcasting state change
17:24:41.623851 Mar 10 RX 93fff700 % KAFKA INTERNAL LOGGER CALLBACK : 1678433081.623 RDKAFKA-7-TERMINATE: kcea#consumer-1: [thrd:cbtest:50066/0]: cbtest:50066/0: Handle is terminating in state INIT: 2 refcnts (0x7fcb940014f8), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf

Adding the stack trace /

Thread 7 (Thread 0x7fcbb5ffa700 (LWP 10568)):
0 0x00007fcbd0ede6dd in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
1 0x00007fcbd0ee93db in thrd_join () from /lib64/libpthread.so.0
2 0x00007fcbd86a2e3e in rd_kafka_destroy_internal (rk=rk@entry=0x7fcbac0036f0) at rdkafka.c:1270
3 0x00007fcbd86a3980 in rd_kafka_thread_main (arg=0x7fcbac0036f0) at rdkafka.c:2120
4 0x00007fcbd0edd2e2 in start_thread () from /lib64/libpthread.so.0
5 0x00007fcbcffffe73 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fcbb71fd700 (LWP 10565)):
0 0x00007fcbd0ede6dd in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
1 0x00007fcbd0ee93db in thrd_join () from /lib64/libpthread.so.0
2 0x00007fcbd869d57c in rd_kafka_destroy_app (rk=0x7fcbac0036f0, flags=, flags@entry=0) at rdkafka.c:1107
3 0x00007fcbd869d907 in rd_kafka_destroy (rk=) at rdkafka.c:1120
4 0x000000000040b15b in CommsKafka::ConsumerClose (this=this@entry=0x7fff33debaa0) at comms_kafka_imp.h:65
5 0x000000000040b55a in CommsKafka::Commit (this=0x7fff33debaa0) at comms_kafka.cc:787
6 0x00000000004461f0 in writerThread (l_void=) at adapter_singleton.h:44
7 0x00007fcbd0edd1da in start_thread () from /lib64/libpthread.so.0
8 0x00007fcbcffffe73 in clone () from /lib64/libc.so.6

But, when we change the broker.version.fallback => 0.9.0 and api.version.request => false in the latest librdkafka version, we are not seeing any issues. The application works fine.

• Could you please advise what we are missing in librdkafka between broker version 0.9.0 and 0.10.0 ?
• Upgrading kakfa => 3.2, librdkafka => 2.0.2 and updating api.version.request => false and broker.version.fallback => 0.9.0. Does this consider a valid upgrade?

Adding the application logs with librdkafka (debug => all) logs.
librdkafka_2_0_2_broker_version_fallback_0_10_0.log
librdkafka_2_0_2_broker_version_fallback_0_9_0.log
Stack_Trace_librdkafka_2_0_2_broker_version_fallback_0_10_0.txt

[karthikeyanajendran1984]

Hi Experts / @edenhill ,

Can some one please reply on this ? this is blocking the librdkafka and Kafka broker upgrade.

Thanks.

Regards,
Karthikeyan Rajendran.

[rpraveensriram]

The above-mentioned issues were resolved for me after updating the generic configuration parameter

allow.auto.create.topics to true