Bugfix in example code #1

bmilekic · 2012-11-01T18:21:24Z

Hey

Have a look at this bugfix. Without it the producer example code corrupts messages in some cases (e.g., when piping output from cat or similar into the example).

The fix I did works, but it's a bit quick. You can pull it in now, or wait until I do some more fixes.

Thank you for open-sourcing your work!

b

…tdin, but then make a copy before passing it into librdkafka for consumption.

… len. This fixes this issue. (reported by: andoma)

edenhill · 2012-11-01T19:02:25Z

Good catch, bmilekic.
The problem is that rd_kafka_produce() does not copy the data (for performance reasons).
So your approach is correct, make an allocated copy (or even better; pre-allocate the fgets() buffer for each spin) and then pass the allocated pointer along with the _FREE flag as you did.

There is no need to null-terminate, rd_kafka_produce() simply sends a binary blob and does not know nor care about strings.

Finalize your fix and I'll merge it.

Thanks.

Bugfix in example code

Fix MSan error

TSan report (founded by ClickHouse CI): Exception: Sanitizer assert found for instance �================== WARNING: ThreadSanitizer: data race (pid=1) Read of size 8 at 0x7b7800127158 by thread T987 (mutexes: read M0, write M1, write M2): #0 __tsan_memcpy <null> (clickhouse+0x74eebb7) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#1 rd_avg_rollover build_docker/./contrib/librdkafka/src/rdavg.h:153:22 (clickhouse+0x1e39753b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#2 rd_kafka_stats_emit_avg build_docker/./contrib/librdkafka/src/rdkafka.c:1354:9 (clickhouse+0x1e39753b) confluentinc#3 rd_kafka_stats_emit_all build_docker/./contrib/librdkafka/src/rdkafka.c:1717:17 (clickhouse+0x1e395c8b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#4 rd_kafka_stats_emit_tmr_cb build_docker/./contrib/librdkafka/src/rdkafka.c:1898:2 (clickhouse+0x1e395c8b) confluentinc#5 rd_kafka_timers_run build_docker/./contrib/librdkafka/src/rdkafka_timer.c:288:4 (clickhouse+0x1e46498a) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#6 rd_kafka_thread_main build_docker/./contrib/librdkafka/src/rdkafka.c:2021:3 (clickhouse+0x1e3919e9) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#7 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1e47a57b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) Previous write of size 8 at 0x7b7800127158 by thread T986: #0 rd_avg_calc build_docker/./contrib/librdkafka/src/rdavg.h:104:38 (clickhouse+0x1e37d71d) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#1 rd_kafka_broker_timeout_scan build_docker/./contrib/librdkafka/src/rdkafka_broker.c:880:25 (clickhouse+0x1e37d71d) confluentinc#2 rd_kafka_broker_ops_io_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:3416:17 (clickhouse+0x1e37d71d) confluentinc#3 rd_kafka_broker_consumer_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:4975:17 (clickhouse+0x1e378e5e) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#4 rd_kafka_broker_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:5080:17 (clickhouse+0x1e378e5e) confluentinc#5 rd_kafka_broker_thread_main build_docker/./contrib/librdkafka/src/rdkafka_broker.c:5237:25 (clickhouse+0x1e372619) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#6 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1e47a57b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) Refs: ClickHouse/ClickHouse#60443

TSan report (founded by ClickHouse CI): Exception: Sanitizer assert found for instance �================== WARNING: ThreadSanitizer: data race (pid=1) Read of size 8 at 0x7b7800127158 by thread T987 (mutexes: read M0, write M1, write M2): #0 __tsan_memcpy <null> (clickhouse+0x74eebb7) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#1 rd_avg_rollover build_docker/./contrib/librdkafka/src/rdavg.h:153:22 (clickhouse+0x1e39753b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#2 rd_kafka_stats_emit_avg build_docker/./contrib/librdkafka/src/rdkafka.c:1354:9 (clickhouse+0x1e39753b) confluentinc#3 rd_kafka_stats_emit_all build_docker/./contrib/librdkafka/src/rdkafka.c:1717:17 (clickhouse+0x1e395c8b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#4 rd_kafka_stats_emit_tmr_cb build_docker/./contrib/librdkafka/src/rdkafka.c:1898:2 (clickhouse+0x1e395c8b) confluentinc#5 rd_kafka_timers_run build_docker/./contrib/librdkafka/src/rdkafka_timer.c:288:4 (clickhouse+0x1e46498a) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#6 rd_kafka_thread_main build_docker/./contrib/librdkafka/src/rdkafka.c:2021:3 (clickhouse+0x1e3919e9) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#7 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1e47a57b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) Previous write of size 8 at 0x7b7800127158 by thread T986: #0 rd_avg_calc build_docker/./contrib/librdkafka/src/rdavg.h:104:38 (clickhouse+0x1e37d71d) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#1 rd_kafka_broker_timeout_scan build_docker/./contrib/librdkafka/src/rdkafka_broker.c:880:25 (clickhouse+0x1e37d71d) confluentinc#2 rd_kafka_broker_ops_io_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:3416:17 (clickhouse+0x1e37d71d) confluentinc#3 rd_kafka_broker_consumer_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:4975:17 (clickhouse+0x1e378e5e) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#4 rd_kafka_broker_serve build_docker/./contrib/librdkafka/src/rdkafka_broker.c:5080:17 (clickhouse+0x1e378e5e) confluentinc#5 rd_kafka_broker_thread_main build_docker/./contrib/librdkafka/src/rdkafka_broker.c:5237:25 (clickhouse+0x1e372619) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) confluentinc#6 _thrd_wrapper_function build_docker/./contrib/librdkafka/src/tinycthread.c:576:9 (clickhouse+0x1e47a57b) (BuildId: 7122171f6a93acda7ea89a6d10cce3ad580a715d) Refs: ClickHouse/ClickHouse#60443 Refs: ClickHouse/ClickHouse#60939 Refs: confluentinc#4625

Bosko Milekic added 2 commits November 1, 2012 11:46

In producer example, use a 2K on-the-stack buffer to read into from s…

bc92100

…tdin, but then make a copy before passing it into librdkafka for consumption.

Comment: strncpy() does not null terminate if it fill all bytes up to…

90dafa4

… len. This fixes this issue. (reported by: andoma)

edenhill added a commit that referenced this pull request Nov 13, 2012

Merge pull request #1 from bmilekic/master

c818ae5

Bugfix in example code

edenhill merged commit c818ae5 into confluentinc:master Nov 13, 2012

xiaojun789 mentioned this pull request Nov 8, 2013

crashes on too large messages #24

Closed

winbatch mentioned this pull request Dec 27, 2013

static linking of librdkafka.a needs dynamic pthreads linking #34

Closed

xianping mentioned this pull request Jan 14, 2014

nginx worker processes exit with signal 11 when publishing messages in nginx #56

Closed

This was referenced Jan 30, 2014

APIv2: Changes in the next version of the API #66

Open

Crash on exit #72

Closed

winbatch mentioned this pull request Mar 18, 2014

rdkafka_performance crashes with -o 'beginning' #92

Closed

FrankVanYoung mentioned this pull request Mar 31, 2014

rkb->rkb_fetching > 0 error #84

Closed

xhcqwlgithub mentioned this pull request Jun 25, 2014

Segmentation fault when creating producer #125

Closed

laxpio mentioned this pull request Aug 4, 2014

Crash on invalid partition in Metadata #132

Closed

desmondhe mentioned this pull request Sep 3, 2014

dead-lock in multi-thread enviroment #145

Closed

winbatch mentioned this pull request Sep 10, 2014

Difference between isr info from librdkafka vs kafka-list-topic.sh #147

Closed

laxpio mentioned this pull request Oct 9, 2014

Crash on rd_kafka_topic_metadata_update #155

Closed

affranchi mentioned this pull request Mar 26, 2015

Seg fault with multi brokers #224

Closed

mahb94 mentioned this pull request Apr 21, 2015

client consume from kafka crushed occasionally #254

Closed

DEvil0000 mentioned this pull request Apr 24, 2015

make tests: test 14 fails for me #259

Closed

edenhill mentioned this pull request Jun 8, 2015

2 races conditions #281

Closed

shalstea mentioned this pull request Jul 1, 2015

rdkafka can't be built with native compilers #319

Closed

rathdeep mentioned this pull request Jul 3, 2015

Best approach to read messages from multiple topics or partitions #322

Closed

chenzhanyiczy mentioned this pull request Jul 7, 2015

Deadlock on broker restart #326

Closed

shalstea mentioned this pull request Jul 14, 2015

librdkafka does not compile cleanly on AIX (using gcc/g++) #328

Closed

sh9189 mentioned this pull request Jul 29, 2015

librdkafka tests crash on solaris #340

Closed

adityanahan mentioned this pull request Aug 8, 2015

Rdkafka Failed to detect broker state #346

Closed

chriscrenshaw mentioned this pull request Aug 14, 2015

rd_kafka_buf_destroy() always aborts - causing an immediate crash of the consumer app #353

Closed

ylgeeker mentioned this pull request Aug 28, 2015

rd_kafka_topic_destroy0 will crash ?? @ 0.8.6 #360

Closed

DEvil0000 mentioned this pull request Oct 12, 2015

rdkafka stopping connection retries on RdKafka::ERR__ALL_BROKERS_DOWN #373

Closed

khpark-git mentioned this pull request Mar 16, 2020

Slowness at the start of the consumer #2756

Closed

7 tasks

bjlvfei mentioned this pull request Apr 29, 2020

failed: err: Local: Broker handle destroyed #2854

Closed

hyhtemple mentioned this pull request Jun 30, 2020

It is crashed ,when use method of the producer in the thread #2947

Closed

dshivashankar1994 mentioned this pull request Jul 6, 2020

rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed - when the subscribed topic is deleted #2963

Closed

harshadvichare mentioned this pull request Jul 17, 2020

Message not delivered if header added. #2996

Closed

3 tasks

waitingF mentioned this pull request Jul 24, 2020

rd_kafka_destroy() hang on #3007

Closed

7 tasks

haolujun mentioned this pull request Jul 28, 2020

Failed to initialize SASL authentication:generic failure (after 0ms in state AUTH_REQ) #3011

Closed

7 tasks

paultomas mentioned this pull request Sep 21, 2020

partition_cb not working with C++ API #2331

Closed

7 tasks

Furuta-Masakazu-quick mentioned this pull request Sep 29, 2020

Producer detects "All broker connections are down" when rolling restarts #3090

Closed

7 tasks

tasneem-hyder mentioned this pull request Oct 13, 2020

~100% CPU usage for rdk:kafka #3107

Closed

7 tasks

azat pushed a commit to azat-archive/librdkafka that referenced this pull request Dec 13, 2020

Merge pull request confluentinc#1 from ClickHouse-Extras/fix-msan

08251b8

Fix MSan error

davinnfld mentioned this pull request Jan 6, 2021

Question: sending single message with no delay #3202

Closed

write2jaydeep mentioned this pull request Mar 3, 2021

consumer failed on broker version fall back #3283

Closed

7 tasks

nagendrababupv mentioned this pull request Mar 21, 2021

Duplicate message from RDKafka #3311

Closed

7 tasks

taj-uddin mentioned this pull request Apr 15, 2021

High CPU usage in librdkafka1.4.0 producer #2986

Closed

7 tasks

johncagle mentioned this pull request Sep 22, 2021

segfault in rd_kafka_msgq_set_metadata() from /lib64/librdkafka.so.1 (corrupt TAILQ) #3559

Open

7 tasks

kenlee1988 mentioned this pull request Nov 18, 2021

rd_kafka_clustid does not return until timeout when kafka has no topic #3620

Open

7 tasks

eelyaj mentioned this pull request Nov 22, 2021

Throughput decrease badly while broker number increase. #3626

Open

7 tasks

yxuechao007 mentioned this pull request Nov 26, 2021

RdKafka::Producer parses aws kafka broker reporting segment errors when the cluster information is retrieved after a few minutes #3632

Open

acoredump mentioned this pull request Dec 15, 2021

coredump in producer::poll #3647

Closed

yuanhongduan mentioned this pull request Dec 27, 2021

librdkafka deadlock at cnd_timedwait when destroy the consumer #3660

Open

benzmarkus mentioned this pull request Oct 19, 2022

Spamming the logs during rebalancing #4028

Open

eig mentioned this pull request Apr 18, 2023

0045-subscribe-update / do_test_regex_many_mock sporadic (but frequent) failure #4259

Open

7 tasks

jbimbert mentioned this pull request Jul 21, 2023

Missing first messages when Producing on Azure with SSL #4363

Open

1 task

mreouven mentioned this pull request Nov 9, 2023

Thread 15 "rdk:broker-1" received signal SIGSEGV, Segmentation fault. #1054 #4505

Open

Webwizzzz mentioned this pull request Mar 20, 2024

Unable to connect C++ consumer client with OAUTHBEARER mechanism to server #4655

Open

chunaiarun mentioned this pull request Aug 29, 2024

timeout issue with async commit , we want to commit every message after consuming #4827

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix in example code #1

Bugfix in example code #1

bmilekic commented Nov 1, 2012

edenhill commented Nov 1, 2012

Bugfix in example code #1

Bugfix in example code #1

Conversation

bmilekic commented Nov 1, 2012

edenhill commented Nov 1, 2012