Segmentation fault when broker restart

[leo-987]

When a broker down and restarted, consumer will core dumped after RebalanceCb:

RebalanceCb: Local: Revoke partitions: test[0], test[1], test[2], test[3], test[4], test[5],
Segmentation fault (core dumped)

[edenhill]

Can you provide a gdb stacktrace?

[leo-987]

RebalanceCb: Local: Revoke partitions: test[0], test[1], test[2], test[3], test[4], test[5], 

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe7fff700 (LWP 9132)]
0x0000000000430b76 in rd_atomic32_get (ra=0x2e74626e2e646825) at rdatomic.h:30
30      return ATOMIC_OP(fetch, add, &ra->val, 0);
Missing separate debuginfos, use: debuginfo-install libgcc-4.8.3-9.el7.x86_64 libstdc++-4.8.3-9.el7.x86_64
(gdb) bt
#0  0x0000000000430b76 in rd_atomic32_get (ra=0x2e74626e2e646825) at rdatomic.h:30
#1  rd_kafka_cgrp_serve (rkcg=0x7fffec0040b0, rkb=rkb@entry=0x7fffec003f90) at rdkafka_cgrp.c:1507
#2  0x00000000004174bc in rd_kafka_broker_serve (rkb=rkb@entry=0x7fffec003f90, timeout_ms=timeout_ms@entry=0) at rdkafka_broker.c:1970
#3  0x0000000000417dcd in rd_kafka_broker_consumer_serve (rkb=0x7fffec003f90) at rdkafka_broker.c:3103
#4  rd_kafka_broker_thread_main (arg=0x7fffec003f90) at rdkafka_broker.c:3191
#5  0x0000000000439229 in _thrd_wrapper_function (aArg=<optimized out>) at tinycthread.c:596
#6  0x00007ffff7bc6df5 in start_thread (arg=0x7fffe7fff700) at pthread_create.c:308
#7  0x00007ffff6ecd1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) frame 1
#1  rd_kafka_cgrp_serve (rkcg=0x7fffec0040b0, rkb=rkb@entry=0x7fffec003f90) at rdkafka_cgrp.c:1507
1507            if (unlikely(rkb->rkb_cgrp != rkcg ||
(gdb) frame 0
#0  0x0000000000430b76 in rd_atomic32_get (ra=0x2e74626e2e646825) at rdatomic.h:30
30      return ATOMIC_OP(fetch, add, &ra->val, 0);
(gdb) frame 2
#2  0x00000000004174bc in rd_kafka_broker_serve (rkb=rkb@entry=0x7fffec003f90, timeout_ms=timeout_ms@entry=0) at rdkafka_broker.c:1970
1970                    rd_kafka_cgrp_serve(rkb->rkb_cgrp, rkb);

[edenhill]

Thanks, will look into it.
Is this the latest master?

[leo-987]

Yes

[edenhill]

How many brokers do you have in your cluster?
How many of them are you taking down?

[leo-987]

3 brokers. One taken down and restart.

[edenhill]

👍

[edenhill]

I believe this was fixed, reopen if reseen.