ZooKeeper integration #4

satoshi75nakamoto · 2013-01-28T23:43:40Z

When is ZooKeeper integration planned?

— Patrick

edenhill · 2013-01-29T10:06:33Z

I want to get 0.8 protocol support out the door (which also simplifies Zookeeper integration somewhat), but after that I'll probably look into it if there is still demand.

Are you in need of ZK support in the producer, consumer, or both?
What would be the preferred version? Kafka 0.7.* or 0.8.*?

satoshi75nakamoto · 2013-01-29T16:19:15Z

Hello Magnus,

I'm looking for ZooKeeper support for the producer. My preferred version would be Kafka 0.8.*.

I'd probably be willing to pay you as a contractor as well. If that helps. :-)

— Patrick

gpatmore · 2014-01-03T18:27:19Z

+1 on zookeeper integration.

edenhill · 2014-01-03T18:59:25Z

@gpatmore I guess you want zookeeper for the consumer, not the producer side, right? I.e., full consumer group balancing, etc.

davinliuda · 2014-04-08T07:38:42Z

+1 on zookeeper integration.

bartw72 · 2014-04-20T11:04:31Z

The integration of zookeeper is not needed to make librdkafka work together with zookeeper. I've submitted a pull request to show how you can add zookeeper api's to your program to add a watch on changing broker lists. This is a slightly modified version of the existing example on how to use librdkafka in the examples directory.

#104

sholsapp · 2014-09-30T22:33:09Z

@edenhill is there already an issue I can watch for full consumer group balancing?

edenhill · 2014-10-01T16:59:54Z

Created issue #152 to track it.

TSan report (founded by ClickHouse CI): Exception: Sanitizer assert found for instance �================== WARNING: ThreadSanitizer: data race (pid=1) Read of size 8 at 0x7b7800127158 by thread T987 (mutexes: read M0, write M1, write M2): #0 __tsan_memcpy <null> (clickhouse+0x74eebb7) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#1 rd_avg_rollover build_docker/./contrib/librdkafka/src/rdavg.h:153:22 (clickhouse+0x1e39753b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#2 rd_kafka_stats_emit_avg build_docker/./contrib/librdkafka/src/rdkafka.c:1354:9 (clickhouse+0x1e39753b) confluentinc#3 rd_kafka_stats_emit_all build_docker/./contrib/librdkafka/src/rdkafka.c:1717:17 (clickhouse+0x1e395c8b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#4 rd_kafka_stats_emit_tmr_cb build_docker/./contrib/librdkafka/src/rdkafka.c:1898:2 (clickhouse+0x1e395c8b) confluentinc#5 rd_kafka_timers_run build_docker/./contrib/librdkafka/src/rdkafka_timer.c:288:4 (clickhouse+0x1e46498a) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#6 rd_kafka_thread_main build_docker/./contrib/librdkafka/src/rdkafka.c:2021:3 (clickhouse+0x1e3919e9) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#7 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1e47a57b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) Previous write of size 8 at 0x7b7800127158 by thread T986: #0 rd_avg_calc build_docker/./contrib/librdkafka/src/rdavg.h:104:38 (clickhouse+0x1e37d71d) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#1 rd_kafka_broker_timeout_scan build_docker/./contrib/librdkafka/src/rdkafka_broker.c:880:25 (clickhouse+0x1e37d71d) confluentinc#2 rd_kafka_broker_ops_io_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:3416:17 (clickhouse+0x1e37d71d) confluentinc#3 rd_kafka_broker_consumer_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:4975:17 (clickhouse+0x1e378e5e) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#4 rd_kafka_broker_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:5080:17 (clickhouse+0x1e378e5e) confluentinc#5 rd_kafka_broker_thread_main build_docker/./contrib/librdkafka/src/rdkafka_broker.c:5237:25 (clickhouse+0x1e372619) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#6 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1e47a57b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) Refs: ClickHouse/ClickHouse#60443

TSan report (founded by ClickHouse CI): Exception: Sanitizer assert found for instance �================== WARNING: ThreadSanitizer: data race (pid=1) Read of size 8 at 0x7b7800127158 by thread T987 (mutexes: read M0, write M1, write M2): #0 __tsan_memcpy <null> (clickhouse+0x74eebb7) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#1 rd_avg_rollover build_docker/./contrib/librdkafka/src/rdavg.h:153:22 (clickhouse+0x1e39753b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#2 rd_kafka_stats_emit_avg build_docker/./contrib/librdkafka/src/rdkafka.c:1354:9 (clickhouse+0x1e39753b) confluentinc#3 rd_kafka_stats_emit_all build_docker/./contrib/librdkafka/src/rdkafka.c:1717:17 (clickhouse+0x1e395c8b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#4 rd_kafka_stats_emit_tmr_cb build_docker/./contrib/librdkafka/src/rdkafka.c:1898:2 (clickhouse+0x1e395c8b) confluentinc#5 rd_kafka_timers_run build_docker/./contrib/librdkafka/src/rdkafka_timer.c:288:4 (clickhouse+0x1e46498a) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#6 rd_kafka_thread_main build_docker/./contrib/librdkafka/src/rdkafka.c:2021:3 (clickhouse+0x1e3919e9) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#7 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1e47a57b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) Previous write of size 8 at 0x7b7800127158 by thread T986: #0 rd_avg_calc build_docker/./contrib/librdkafka/src/rdavg.h:104:38 (clickhouse+0x1e37d71d) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#1 rd_kafka_broker_timeout_scan build_docker/./contrib/librdkafka/src/rdkafka_broker.c:880:25 (clickhouse+0x1e37d71d) confluentinc#2 rd_kafka_broker_ops_io_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:3416:17 (clickhouse+0x1e37d71d) confluentinc#3 rd_kafka_broker_consumer_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:4975:17 (clickhouse+0x1e378e5e) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#4 rd_kafka_broker_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:5080:17 (clickhouse+0x1e378e5e) confluentinc#5 rd_kafka_broker_thread_main build_docker/./contrib/librdkafka/src/rdkafka_broker.c:5237:25 (clickhouse+0x1e372619) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#6 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1e47a57b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) Refs: ClickHouse/ClickHouse#60443 Refs: ClickHouse/ClickHouse#60939 Refs: confluentinc#4625

xiaojun789 mentioned this issue Nov 8, 2013

crashes on too large messages #24

Closed

winbatch mentioned this issue Dec 27, 2013

static linking of librdkafka.a needs dynamic pthreads linking #34

Closed

winbatch mentioned this issue Feb 10, 2014

Crash on exit #72

Closed

winbatch mentioned this issue Mar 18, 2014

rdkafka_performance crashes with -o 'beginning' #92

Closed

FrankVanYoung mentioned this issue Mar 31, 2014

rkb->rkb_fetching > 0 error #84

Closed

xhcqwlgithub mentioned this issue Jun 25, 2014

Segmentation fault when creating producer #125

Closed

laxpio mentioned this issue Aug 4, 2014

Crash on invalid partition in Metadata #132

Closed

desmondhe mentioned this issue Sep 3, 2014

dead-lock in multi-thread enviroment #145

Closed

winbatch mentioned this issue Sep 10, 2014

Difference between isr info from librdkafka vs kafka-list-topic.sh #147

Closed

laxpio mentioned this issue Oct 9, 2014

Crash on rd_kafka_topic_metadata_update #155

Closed

This was referenced Feb 13, 2015

offset & consumer groups #199

Closed

(local) consumer groups: not working #202

Closed

affranchi mentioned this issue Mar 26, 2015

Seg fault with multi brokers #224

Closed

mahb94 mentioned this issue Apr 21, 2015

client consume from kafka crushed occasionally #254

Closed

DEvil0000 mentioned this issue Apr 24, 2015

make tests: test 14 fails for me #259

Closed

edenhill added the consumer label May 15, 2015

chenzhanyiczy mentioned this issue Jul 7, 2015

Deadlock on broker restart #326

Closed

shalstea mentioned this issue Jul 13, 2015

rdkafka can't be built with native compilers #319

Closed

sh9189 mentioned this issue Jul 29, 2015

librdkafka tests crash on solaris #340

Closed

adityanahan mentioned this issue Aug 8, 2015

Rdkafka Failed to detect broker state #346

Closed

chriscrenshaw mentioned this issue Aug 14, 2015

rd_kafka_buf_destroy() always aborts - causing an immediate crash of the consumer app #353

Closed

ylgeeker mentioned this issue Aug 28, 2015

rd_kafka_topic_destroy0 will crash ?? @ 0.8.6 #360

Closed

pradeepp2019 mentioned this issue Apr 23, 2019

Query on Async programming (Seastar) with librdkafka #2292

Closed

write2jaydeep mentioned this issue May 9, 2019

multiple call of rd_kafka_new() from threads #2317

Closed

6 tasks

sezruby mentioned this issue May 27, 2019

rd_kafka_destroy_app got stuck in pthread_join #2339

Closed

7 tasks

jcalcote mentioned this issue Jun 28, 2019

assert fail in #2384

Closed

7 tasks

MockingJayWong mentioned this issue Jul 9, 2019

rd_kafka_topic_new after using admin api #2399

Closed

7 tasks

better2being mentioned this issue Jul 29, 2019

Producer with .net cannot write to kafka cluster #2437

Closed

2 tasks

MockingJayWong mentioned this issue Aug 2, 2019

rd_kafka_toppar_enq_msg() lock when produce msg #2447

Closed

7 tasks

sarkanyi mentioned this issue Sep 9, 2019

Recovery takes too long after stopping and restarting a broker in a cluster #2508

Closed

3 tasks

hyhtemple mentioned this issue Oct 14, 2019

use the produce of the librdkafka c++ interface ,the program report "Program terminated with signal 6, Aborted" #2571

Closed

hyhtemple mentioned this issue Oct 22, 2019

the program of the Consumer crashed when create topic #2579

Closed

aelanthi mentioned this issue Dec 16, 2019

confluent python kafka client still crashes after 1.0.1 upgrade #2658

Closed

7 tasks

hyhtemple mentioned this issue Jun 30, 2020

It is crashed ,when use method of the producer in the thread #2947

Closed

dshivashankar1994 mentioned this issue Jul 6, 2020

rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed - when the subscribed topic is deleted #2963

Closed

waitingF mentioned this issue Jul 24, 2020

rd_kafka_destroy() hang on #3007

Closed

7 tasks

Furuta-Masakazu-quick mentioned this issue Sep 29, 2020

Producer detects "All broker connections are down" when rolling restarts #3090

Closed

7 tasks

nagendrababupv mentioned this issue Mar 21, 2021

Duplicate message from RDKafka #3311

Closed

7 tasks

taj-uddin mentioned this issue Apr 15, 2021

High CPU usage in librdkafka1.4.0 producer #2986

Closed

7 tasks

johncagle mentioned this issue Sep 22, 2021

segfault in rd_kafka_msgq_set_metadata() from /lib64/librdkafka.so.1 (corrupt TAILQ) #3559

Open

7 tasks

eelyaj mentioned this issue Nov 22, 2021

Throughput decrease badly while broker number increase. #3626

Open

7 tasks

yxuechao007 mentioned this issue Nov 26, 2021

RdKafka::Producer parses aws kafka broker reporting segment errors when the cluster information is retrieved after a few minutes #3632

Open

acoredump mentioned this issue Dec 15, 2021

coredump in producer::poll #3647

Closed

yuanhongduan mentioned this issue Dec 27, 2021

librdkafka deadlock at cnd_timedwait when destroy the consumer #3660

Open

eig mentioned this issue Apr 18, 2023

0045-subscribe-update / do_test_regex_many_mock sporadic (but frequent) failure #4259

Open

7 tasks

mreouven mentioned this issue Nov 9, 2023

Thread 15 "rdk:broker-1" received signal SIGSEGV, Segmentation fault. #1054 #4505

Open

chunaiarun mentioned this issue Aug 29, 2024

timeout issue with async commit , we want to commit every message after consuming #4827

Closed

thmic mentioned this issue Nov 29, 2024

segmentation fault from rdkafka_topic.c #4907

Open

7 tasks

thilinak mentioned this issue Dec 12, 2024

C++ application built with librdkafka with GCC TSAN crashes when starting #4929

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZooKeeper integration #4

ZooKeeper integration #4

satoshi75nakamoto commented Jan 28, 2013

edenhill commented Jan 29, 2013

satoshi75nakamoto commented Jan 29, 2013

gpatmore commented Jan 3, 2014

edenhill commented Jan 3, 2014

davinliuda commented Apr 8, 2014

bartw72 commented Apr 20, 2014

sholsapp commented Sep 30, 2014

edenhill commented Oct 1, 2014

ZooKeeper integration #4

ZooKeeper integration #4

Comments

satoshi75nakamoto commented Jan 28, 2013

edenhill commented Jan 29, 2013

satoshi75nakamoto commented Jan 29, 2013

gpatmore commented Jan 3, 2014

edenhill commented Jan 3, 2014

davinliuda commented Apr 8, 2014

bartw72 commented Apr 20, 2014

sholsapp commented Sep 30, 2014

edenhill commented Oct 1, 2014