Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiple call of rd_kafka_new() from threads #2317

Closed
5 of 6 tasks
write2jaydeep opened this issue May 8, 2019 · 5 comments
Closed
5 of 6 tasks

multiple call of rd_kafka_new() from threads #2317

write2jaydeep opened this issue May 8, 2019 · 5 comments

Comments

@write2jaydeep
Copy link

Description

Hi @edenhill
I am calling rd_kafka_new() from multiple threads, does it create any problem from internal implementation point of view??

currently, my main application is getting crashed at the call of rd_kafka_new().
the crash may not be relevant to librdkafka

logs

G 112100 -0-KAFKA-INT: 1557328860.270 ARI-RDKAFKA-7-SASL: rdkafka#producer-1: [thrd:app]: Selected provider Cyrus for SASL mechani
sm GSSAPI
G 112100 -0-KAFKA-INT: 1557328860.270 ARI-RDKAFKA-7-OPENSSL: rdkafka#producer-1: [thrd:app]: librdkafka built with OpenSSL version
0x1000109f
G 112100 -0-KAFKA-INT: 1557328860.279 ARI-RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading CA certificate(s) from file certs
/ca-cert
G 112100 -0-KAFKA-INT: 1557328860.281 ARI-RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading certificate from file /Stratusoft
/ARI_Techsupport/SGI_CLIENT_BUILD/sgi_client_instl/certs/rdclient.pem
G 112100 -0-KAFKA-INT: 1557328860.281 ARI-RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading private key file from /Stratusoft
/ARI_Techsupport/SGI_CLIENT_BUILD/sgi_client_instl/certs/rdclient.key
G 112100 -0-KAFKA-INT: 1557328860.282 ARI-RDKAFKA-7-SSLPASSWD: rdkafka#producer-1: [thrd:app]: Private key file "/Stratusoft/ARI_T
echsupport/SGI_CLIENT_BUILD/sgi_client_instl/certs/rdclient.key" requires password
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-WAKEUPFD: rdkafka#producer-1: [thrd:app]: sasl_ssl://mydomain.com:9997
/bootstrap: Enabled low-latency ops queue wake-ups
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-BROKER: rdkafka#producer-1: [thrd:app]: sasl_ssl://mydomain.com:9997/b
ootstrap: Added new broker with NodeId -1
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:app]: sasl_ssl://mydomain.com:9997/
bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt
(s))
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-BRKMAIN: rdkafka#producer-1: [thrd::0/internal]: :0/internal: Enter main broke
r thread
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-BRKMAIN: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Enter main broker thread
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Received CONNECT op
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-STATE: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstra
p]: sasl_ssl://mydomain.com:9997/bootstrap: Broker changed state INIT -> TRY_CONNECT
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-BROADCAST: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/boot
strap]: Broadcasting state change
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: broker in state TRY_CONNECT connecting
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-STATE: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstra
p]: sasl_ssl://mydomain.com:9997/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
G 112100 -0-KAFKA-INT: 1557328860.286 ARI-RDKAFKA-7-BROADCAST: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/boot
strap]: Broadcasting state change
G 112100 -0-KAFKA-INT: 1557328860.288 ARI-RDKAFKA-7-INIT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap
]: librdkafka v1.0.0 (0x10000ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sa
sl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins, GCC GXX INSTALL GNULD LIBDL PLUGINS ZLIB SSL
SASL_CYRUS HDRHISTOGRAM SNAPPY SOCKEM SASL_SCRAM, debug 0xffff)
G 112100 -1-LOG: All properties has been set successfully...#2!
G 112100 -0-KAFKA-INT: 1557328860.289 ARI-RDKAFKA-7-SASL: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap
]: Selected provider Cyrus for SASL mechanism GSSAPI
G 112100 -0-KAFKA-INT: 1557328860.289 ARI-RDKAFKA-7-OPENSSL: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: librdkafka built with OpenSSL version 0x1000109f
G 112100 -0-KAFKA-INT: 1557328860.290 ARI-RDKAFKA-7-SSL: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]
: Loading CA certificate(s) from file certs/ca-cert
G 112100 -0-KAFKA-INT: 1557328860.291 ARI-RDKAFKA-7-SSL: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]
: Loading certificate from file /Stratusoft/ARI_Techsupport/SGI_CLIENT_BUILD/sgi_client_instl/certs/
rdclient.pem
G 112100 -0-KAFKA-INT: 1557328860.291 ARI-RDKAFKA-7-SSL: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]
: Loading private key file from /Stratusoft/ARI_Techsupport/SGI_CLIENT_BUILD/sgi_client_instl/certs/
rdclient.key
G 112100 -0-KAFKA-INT: 1557328860.292 ARI-RDKAFKA-7-SSLPASSWD: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/boot
strap]: Private key file "/Stratusoft/ARI_Techsupport/SGI_CLIENT_BUILD/sgi_client_instl/certs/rdclie
nt.key" requires password
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-WAKEUPFD: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/boots
trap]: sasl_ssl://mydomain.com:9997/bootstrap: Enabled low-latency ops queue wake-ups
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-BROKER: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstr
ap]: sasl_ssl://mydomain.com:9997/bootstrap: Added new broker with NodeId -1
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Selected for cluster connection: bootstrap ser
vers added (broker has 0 connection attempt(s))
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-BRKMAIN: rdkafka#producer-2: [thrd::0/internal]: :0/internal: Enter main broke
r thread
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-BRKMAIN: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Enter main broker thread
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Received CONNECT op
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-STATE: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstra
p]: sasl_ssl://mydomain.com:9997/bootstrap: Broker changed state INIT -> TRY_CONNECT
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-BROADCAST: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/boot
strap]: Broadcasting state change
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: broker in state TRY_CONNECT connecting
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-STATE: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstra
p]: sasl_ssl://mydomain.com:9997/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
G 112100 -0-KAFKA-INT: 1557328860.296 ARI-RDKAFKA-7-BROADCAST: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/boot
strap]: Broadcasting state change
G 112100 -0-KAFKA-INT: 1557328860.298 ARI-RDKAFKA-7-INIT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap
]: librdkafka v1.0.0 (0x10000ff) rdkafka#producer-2 initialized (builtin.features gzip,snappy,ssl,sa
sl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins, GCC GXX INSTALL GNULD LIBDL PLUGINS ZLIB SSL
SASL_CYRUS HDRHISTOGRAM SNAPPY SOCKEM SASL_SCRAM, debug 0xffff)
G 112100 -0-KAFKA-INT: 1557328860.304 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Connecting to ipv4#xx.xx.xx.xx:9997 (sasl_ssl
) with socket 15
G 112100 -0-KAFKA-INT: 1557328860.307 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Connecting to ipv4#xx.xx.xx.xx:9997 (sasl_ssl
) with socket 10
G 112100 -0-KAFKA-INT: 1557328860.331 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Connected to ipv4#xx.xx.xx.xx:9997
G 112100 -0-KAFKA-INT: 1557328860.359 ARI-RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootst
rap]: sasl_ssl://mydomain.com:9997/bootstrap: Connected to ipv4#xx.xx.xx.xx:9997

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

  • librdkafka version (release number or git tag): 1.0.0
  • Apache Kafka version: 1.1.1
  • librdkafka client configuration: edit librdkafka.properties and set following properties with ';' separated metadata.broker.list=SASL_SSL://BROKER_FQDN:PORT builtin.features=sasl_gssapi security.protocol=sasl_ssl sasl.mechanisms=GSSAPI sasl.kerberos.service.name=(KAFKASERVICE) sasl.kerberos.principal=(KAFKASERVICE)/(BROKER_FQDN)@(REALM_FQDN) sasl.kerberos.keytab=(FULL_PATH_NAME)/certs/stratusoft1.keytab ssl.ca.location=(FULL_PATH_NAME)/certs/ca-cert ssl.certificate.location=(FULL_PATH_NAME)/certs/rdclient.pem ssl.key.location=(FULL_PATH_NAME)/certs/rdclient.key ssl.key.password=YOUR_PASSWORD
  • Operating system: Linux like
  • [] Provide logs (with debug=.. as necessary) from librdkafka
  • Provide broker log excerpts
  • Critical issue
@edenhill
Copy link
Contributor

edenhill commented May 8, 2019

Should be fine, librdkafka's test suite instantiates multiple rd_kafka_t instances from multiple threads.

Use valgrind or asan to figure out what is wrong.

@write2jaydeep
Copy link
Author

Hi, I tried it with the sample example.
I don't destroy the previous object. I mean, I am trying to create multiple rd_kafka_t instances without destroying the previous one. it crashes.

here is the log of Valgrind ( I never used valgrind before. pardon me if I did something wrong)

==5095== Memcheck, a memory error detector
==5095== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==5095== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==5095== Command: ./start_pr_ke_ssl.sh
==5095==
--5095-- Valgrind options:
--5095-- --leak-check=full
--5095-- -v
--5095-- Contents of /proc/version:
--5095-- Linux version 4.15.0-1029-gcp (buildd@lgw01-amd64-006) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu116.04.10)) #3116.04.1-Ubuntu SMP Fri Mar 22 13:06:42 UTC 2019
--5095--
--5095-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-lzcnt-rdtscp-sse3-avx-avx2-bmi
--5095-- Page sizes: currently 4096, max supported 4096
--5095-- Valgrind library directory: /usr/lib/valgrind
--5095-- Reading syms from /bin/dash
--5095-- object doesn't have a symbol table
--5095-- Reading syms from /lib/x86_64-linux-gnu/ld-2.23.so
--5095-- Considering /lib/x86_64-linux-gnu/ld-2.23.so ..
--5095-- .. CRC mismatch (computed 10d8ea02 wanted 4fdda1aa)
--5095-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/ld-2.23.so ..
--5095-- .. CRC is valid
--5095-- Reading syms from /usr/lib/valgrind/memcheck-amd64-linux
--5095-- Considering /usr/lib/valgrind/memcheck-amd64-linux ..
--5095-- .. CRC mismatch (computed eea41ea9 wanted 2009db78)
--5095-- object doesn't have a symbol table
--5095-- object doesn't have a dynamic symbol table
--5095-- Scheduler: using generic scheduler lock implementation.
--5095-- Reading suppressions file: /usr/lib/valgrind/default.supp
==5095== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-5095-by-root-on-???
==5095== embedded gdbserver: writing to /tmp/vgdb-pipe-to-vgdb-from-5095-by-root-on-???
==5095== embedded gdbserver: shared mem /tmp/vgdb-pipe-shared-mem-vgdb-5095-by-root-on-???
==5095==
==5095== TO CONTROL THIS PROCESS USING vgdb (which you probably
==5095== don't want to do, unless you know exactly what you're doing,
==5095== or are doing some strange experiment):
==5095== /usr/lib/valgrind/../../bin/vgdb --pid=5095 ...command...
==5095==
==5095== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==5095== /path/to/gdb ./start_pr_ke_ssl.sh
==5095== and then give GDB the following command
==5095== target remote | /usr/lib/valgrind/../../bin/vgdb --pid=5095
==5095== --pid is optional if only one valgrind process is running
==5095==
--5095-- REDIR: 0x401cf10 (ld-linux-x86-64.so.2:strlen) redirected to 0x3809e181 (???)
--5095-- Reading syms from /usr/lib/valgrind/vgpreload_core-amd64-linux.so
--5095-- Considering /usr/lib/valgrind/vgpreload_core-amd64-linux.so ..
--5095-- .. CRC mismatch (computed 2567ccf6 wanted 49420590)
--5095-- object doesn't have a symbol table
--5095-- Reading syms from /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so
--5095-- Considering /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so ..
--5095-- .. CRC mismatch (computed 0e27c9a8 wanted ac585421)
--5095-- object doesn't have a symbol table
==5095== WARNING: new redirection conflicts with existing -- ignoring it
--5095-- old: 0x0401cf10 (strlen ) R-> (0000.0) 0x3809e181 ???
--5095-- new: 0x0401cf10 (strlen ) R-> (2007.0) 0x04c31020 strlen
--5095-- REDIR: 0x401b860 (ld-linux-x86-64.so.2:index) redirected to 0x4c30bc0 (index)
--5095-- REDIR: 0x401ba80 (ld-linux-x86-64.so.2:strcmp) redirected to 0x4c320d0 (strcmp)
--5095-- REDIR: 0x401dc70 (ld-linux-x86-64.so.2:mempcpy) redirected to 0x4c35270 (mempcpy)
--5095-- Reading syms from /lib/x86_64-linux-gnu/libc-2.23.so
--5095-- Considering /lib/x86_64-linux-gnu/libc-2.23.so ..
--5095-- .. CRC mismatch (computed 10016417 wanted 1f868c97)
--5095-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.23.so ..
--5095-- .. CRC is valid
--5095-- REDIR: 0x4ec9a00 (libc.so.6:strcasecmp) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec5280 (libc.so.6:strcspn) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ecbcf0 (libc.so.6:strncasecmp) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec76f0 (libc.so.6:strpbrk) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec7a80 (libc.so.6:strspn) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec914b (libc.so.6:memcpy@GLIBC_2.2.5) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec5160 (libc.so.6:strcpy) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec9850 (libc.so.6:stpcpy) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec3a80 (libc.so.6:index) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec8bb0 (libc.so.6:bcmp) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec3cd0 (libc.so.6:strcmp) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ece3f0 (libc.so.6:memcpy@@GLIBC_2.14) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec9330 (libc.so.6:mempcpy) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec8630 (libc.so.6:strstr) redirected to 0x4a286f0 (_vgnU_ifunc_wrapper)
--5095-- REDIR: 0x4ec7400 (libc.so.6:rindex) redirected to 0x4c308a0 (rindex)
--5095-- REDIR: 0x4ebe130 (libc.so.6:malloc) redirected to 0x4c2db20 (malloc)
--5095-- REDIR: 0x4ed0760 (libc.so.6:strchrnul) redirected to 0x4c34da0 (strchrnul)
--5095-- REDIR: 0x4ec5720 (libc.so.6:strlen) redirected to 0x4c30f60 (strlen)
--5095-- REDIR: 0x4f88240 (libc.so.6:__mempcpy_avx_unaligned) redirected to 0x4c34eb0 (mempcpy)
--5095-- REDIR: 0x4ece470 (libc.so.6:__GI_memcpy) redirected to 0x4c32b00 (__GI_memcpy)
--5095-- REDIR: 0x4ed9570 (libc.so.6:__strcmp_sse2_unaligned) redirected to 0x4c31f90 (strcmp)
--5095-- REDIR: 0x4f87ea0 (libc.so.6:__memcpy_avx_unaligned) redirected to 0x4c324a0 (memcpy@@GLIBC_2.14)
--5095-- REDIR: 0x4f845f0 (libc.so.6:__strcspn_sse42) redirected to 0x4c35610 (strcspn)
--5095-- REDIR: 0x4f84730 (libc.so.6:__strpbrk_sse42) redirected to 0x4c355b0 (strpbrk)
--5095-- REDIR: 0x4fa8430 (libc.so.6:__memcmp_sse4_1) redirected to 0x4c33cd0 (__memcmp_sse4_1)
--5095-- REDIR: 0x4ec3ab0 (libc.so.6:__GI_strchr) redirected to 0x4c30a00 (__GI_strchr)
--5095-- REDIR: 0x4ebe6c0 (libc.so.6:realloc) redirected to 0x4c2fce0 (realloc)
1557399218.676 RDKAFKA-7-SASL: rdkafka#producer-1: [thrd:app]: Selected provider Cyrus for SASL mechanism GSSAPI
1557399218.676 RDKAFKA-7-OPENSSL: rdkafka#producer-1: [thrd:app]: librdkafka built with OpenSSL version 0x1000207f
1557399218.676 RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading CA certificate(s) from file ca-cert
1557399218.676 RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading certificate from file rdclient.pem
1557399218.677 RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading private key file from rdclient.key
1557399218.677 RDKAFKA-7-SSLPASSWD: rdkafka#producer-1: [thrd:app]: Private key file "rdclient.key" requires password
1557399218.677 RDKAFKA-7-WAKEUPFD: rdkafka#producer-1: [thrd:app]: sasl_ssl://mydomain.com:9997/bootstrap: Enabled low-latency ops queue wake-ups
1557399218.677 RDKAFKA-7-BROKER: rdkafka#producer-1: [thrd:app]: sasl_ssl://mydomain.com:9997/bootstrap: Added new broker with NodeId -1
1557399218.677 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:app]: sasl_ssl://mydomain.com:9997/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
1557399218.677 RDKAFKA-7-INIT: rdkafka#producer-1: [thrd:app]: librdkafka v1.0.0-RC5-9-g8c5c46-dirty (0x10000ff) rdkafka#producer-1 initialized (builtin.features 0xffff, debug 0xffff)
1557399218.677 RDKAFKA-7-SASL: rdkafka#producer-2: [thrd:app]: Selected provider Cyrus for SASL mechanism GSSAPI
1557399218.677 RDKAFKA-7-OPENSSL: rdkafka#producer-2: [thrd:app]: librdkafka built with OpenSSL version 0x1000207f
1557399218.678 RDKAFKA-7-SSL: rdkafka#producer-2: [thrd:app]: Loading CA certificate(s) from file ca-cert
1557399218.678 RDKAFKA-7-SSL: rdkafka#producer-2: [thrd:app]: Loading certificate from file rdclient.pem
1557399218.678 RDKAFKA-7-SSL: rdkafka#producer-2: [thrd:app]: Loading private key file from rdclient.key
1557399218.678 RDKAFKA-7-SSLPASSWD: rdkafka#producer-2: [thrd:app]: Private key file "rdclient.key" requires password
1557399218.678 RDKAFKA-7-WAKEUPFD: rdkafka#producer-2: [thrd:app]: sasl_ssl://mydomain.com:9997/bootstrap: Enabled low-latency ops queue wake-ups
1557399218.678 RDKAFKA-7-BROKER: rdkafka#producer-2: [thrd:app]: sasl_ssl://mydomain.com:9997/bootstrap: Added new broker with NodeId -1
1557399218.678 RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:app]: sasl_ssl://mydomain.com:9997/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
1557399218.678 RDKAFKA-7-INIT: rdkafka#producer-2: [thrd:app]: librdkafka v1.0.0-RC5-9-g8c5c46-dirty (0x10000ff) rdkafka#producer-2 initialized (builtin.features 0xffff, debug 0xffff)
1557399218.678 RDKAFKA-7-TOPIC: rdkafka#producer-1: [thrd:app]: New local topic: ssl_test
1557399218.678 RDKAFKA-7-TOPPARNEW: rdkafka#producer-1: [thrd:app]: NEW ssl_test [-1] 0x1b5b220 (at rd_kafka_topic_new0:393)
1557399218.678 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: leader query
1557399218.678 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: leader query
1557399218.678 RDKAFKA-7-METADATA: rdkafka#producer-1: [thrd:app]: Skipping metadata refresh of 1 topic(s): no usable brokers
% Type stuff and hit enter to send
1557399218.679 RDKAFKA-7-BRKMAIN: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Enter main broker thread
1557399218.679 RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Received CONNECT op
1557399218.679 RDKAFKA-7-STATE: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Broker changed state INIT -> TRY_CONNECT
1557399218.679 RDKAFKA-7-BROADCAST: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: Broadcasting state change
1557399218.679 RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: broker in state TRY_CONNECT connecting
1557399218.679 RDKAFKA-7-STATE: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
1557399218.679 RDKAFKA-7-BROADCAST: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: Broadcasting state change
1557399218.679 RDKAFKA-7-BRKMAIN: rdkafka#producer-2: [thrd::0/internal]: :0/internal: Enter main broker thread
1557399218.679 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Enter main broker thread
1557399218.679 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Received CONNECT op
1557399218.679 RDKAFKA-7-STATE: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Broker changed state INIT -> TRY_CONNECT
1557399218.679 RDKAFKA-7-BROADCAST: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: Broadcasting state change
1557399218.680 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: broker in state TRY_CONNECT connecting
1557399218.680 RDKAFKA-7-STATE: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
1557399218.680 RDKAFKA-7-BROADCAST: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: Broadcasting state change
1557399218.680 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: [thrd::0/internal]: :0/internal: Enter main broker thread
1557399218.695 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Connecting to ipv4#xx.xx.xx.xx:9997 (sasl_ssl) with socket 13
1557399218.695 RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Connecting to ipv4#xx.xx.xx.xx:9997 (sasl_ssl) with socket 12
1557399218.696 RDKAFKA-7-CONNECT: rdkafka#producer-2: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Connected to ipv4#xx.xx.xx.xx:9997
1557399218.696 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://mydomain.com:9997/bootstrap]: sasl_ssl://mydomain.com:9997/bootstrap: Connected to ipv4#xx.xx.xx.xx:9997
--5095-- REDIR: 0x4ec9890 (libc.so.6:__GI_stpcpy) redirected to 0x4c33f80 (__GI_stpcpy)
--5095-- REDIR: 0x4ec93b0 (libc.so.6:__GI_mempcpy) redirected to 0x4c34fa0 (__GI_mempcpy)
Segmentation fault (core dumped)
--5095-- REDIR: 0x4ebe4f0 (libc.so.6:free) redirected to 0x4c2ed80 (free)
==5095==
==5095== HEAP SUMMARY:
==5095== in use at exit: 1,041 bytes in 29 blocks
==5095== total heap usage: 38 allocs, 9 frees, 5,153 bytes allocated
==5095==
==5095== Searching for pointers to 29 not-freed blocks
==5095== Checked 69,664 bytes
==5095==
==5095== LEAK SUMMARY:
==5095== definitely lost: 0 bytes in 0 blocks
==5095== indirectly lost: 0 bytes in 0 blocks
==5095== possibly lost: 0 bytes in 0 blocks
==5095== still reachable: 1,041 bytes in 29 blocks
==5095== suppressed: 0 bytes in 0 blocks
==5095== Reachable blocks (those to which a pointer was found) are not shown.
==5095== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==5095==
==5095== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==5095== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

@edenhill
Copy link
Contributor

edenhill commented May 9, 2019

For some reason valgrind doesn't show the stacktrace of the crash.

How about you run the program as usual and have it generate a core file, then open that corefile in gdb and print the backtrace from there instead, might give you some ideas what is wrong.

@write2jaydeep
Copy link
Author

Thank you! I increased ulimit, now it creates a core file.

(gdb) bt
#0 _int_malloc (av=av@entry=0x7ffff69bcb20 <main_arena>, bytes=bytes@entry=904) at malloc.c:3516
#1 0x00007ffff667cdca in __libc_calloc (n=, elem_size=) at malloc.c:3236
#2 0x000000000042ad10 in rd_calloc (sz=904, num=1) at rd.h:94
#3 rd_kafka_topic_new0 (rk=rk@entry=0x71cd60, topic=topic@entry=0x7fffffffe45a "ssl_test", conf=conf@entry=0x6fce10, existing=existing@entry=0x7fffffffd4c4,
do_lock=do_lock@entry=1) at rdkafka_topic.c:277
#4 0x000000000042bf9f in rd_kafka_topic_new (rk=0x71cd60, topic=topic@entry=0x7fffffffe45a "ssl_test", conf=conf@entry=0x6fce10) at rdkafka_topic.c:427
#5 0x0000000000405787 in main (argc=36, argv=0x7fffffffe0d8) at rdkafka_example.c:719
(gdb)

@edenhill
Copy link
Contributor

edenhill commented May 9, 2019

That's the libc allocator failing, which is usually indication of user-after-free or memory corruption.
You'll need to find the source of this problem using valgrind or asan, it is most likely not a problem in librdkafka.

@edenhill edenhill closed this as completed May 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@edenhill @write2jaydeep