BROKERFAIL : failed: err: Local: Bad message format: (errno: Resource temporarily unavailable) #1682

senthilec566 · 2018-02-13T13:18:48Z

Description

BROKERFAIL : failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)

How to reproduce

Produce messages at rate 5MB/sec or 10MB/sec.

Checklist

Please provide the following information:

librdkafka version (release number or git tag): v0.11.3
Apache Kafka version: Kafka version : 0.11.0.0 & Kafka commitId : cb8625948210849f
librdkafka client configuration: `message.max.bytes=67108864,socket.timeout.ms=10000,fetch.message.max.bytes=67108864,receive.message.max.bytes=67108864,max.partition.fetch.bytes=67108864,queue.buffering.max.ms=100,message.copy.max.bytes=67108864,broker.version.fallback=0.11.0,compression.codec=lz4,auto.commit.enable=true,auto.commit.interval.ms=1000,debug=protocol
Operating system: Ubuntu 16.04.2 LTS
Provide logs (with debug=.. as necessary) from librdkafka

 102 HighLevelKafkaConsumer.cc:160] Assigning partitions
I0213 12:36:28.368705   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Sent OffsetFetchRequest (v1, 417 bytes @ 0, CorrId 8)
I0213 12:36:28.369091   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RECV, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Received OffsetFetchResponse (v1, 1035 bytes, CorrId 8, rtt 0.37ms)
I0213 12:36:28.369307   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Sent FetchRequest (v4, 623 bytes @ 0, CorrId 9)
I0213 12:36:28.369307   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent FetchRequest (v4, 623 bytes @ 0, CorrId 5)
I0213 12:36:28.369808   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROKERFAIL, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)
I0213 12:36:28.369879   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RETRY, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Retrying FetchRequest (v4, 623 bytes, retry 1/2)
E0213 12:36:28.369889   102 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0213 12:36:28.369941   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent MetadataRequest (v2, 118 bytes @ 0, CorrId 6)
I0213 12:36:28.370074   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROKERFAIL, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)
I0213 12:36:28.370117   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RETRY, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Retrying FetchRequest (v4, 623 bytes, retry 1/2)
E0213 12:36:28.370117   102 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0213 12:36:28.370136   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RETRY, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Retrying MetadataRequest (v2, 118 bytes, retry 1/2)
E0213 12:36:28.370169   102 HighLevelKafkaBase.cc:76] All kafka brokers down. Sleeping for: 1ms. 
E0213 12:36:28.371233   102 HighLevelKafkaBase.cc:80] Retries left: 10
I0213 12:36:28.470257   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: CONNECTED, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Connected (#2)
I0213 12:36:28.470307   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Sent ApiVersionRequest (v0, 55 bytes @ 0, CorrId 10)
I0213 12:36:28.470471   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: CONNECTED, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Connected (#2)
I0213 12:36:28.470515   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent ApiVersionRequest (v0, 55 bytes @ 0, CorrId 7)
I0213 12:36:28.470783   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RECV, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Received ApiVersionResponse (v0, 210 bytes, CorrId 10, rtt 0.46ms)
I0213 12:36:28.470824   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Sent FetchRequest (v4, 623 bytes @ 0, CorrId 11)
I0213 12:36:28.471040   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RECV, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Received ApiVersionResponse (v0, 210 bytes, CorrId 7, rtt 0.51ms)
I0213 12:36:28.471089   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent FetchRequest (v4, 623 bytes @ 0, CorrId 8)
I0213 12:36:28.471101   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent MetadataRequest (v2, 118 bytes @ 0, CorrId 9)
I0213 12:36:28.471364   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROKERFAIL, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)
I0213 12:36:28.471395   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RETRY, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Retrying FetchRequest (v4, 623 bytes, retry 2/2)
E0213 12:36:28.471431   102 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format


102 HighLevelKafkaBase.cc:80] Retries left: 6
E0213 12:56:44.859793   102 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0213 12:56:44.859834   102 HighLevelKafkaBase.cc:76] All kafka brokers down. Sleeping for: 313ms. 
I0213 12:56:44.915302   107 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 4, fac: COMMITFAIL, event: [thrd:main]: Offset commit (cgrp auto commit timer) failed for 60/60 partition(s): Local: Waiting for coordinator: logs.raw[0]@353506, logs.raw[1], logs.raw[2], logs.raw[3], logs.raw[4], logs.raw[5], logs.raw[6], logs.raw[7], logs.raw[8], logs.raw[9], logs.raw[10], logs.raw[11], logs.raw[12], logs.in..
E0213 12:56:45.172904   102 HighLevelKafkaBase.cc:80] Retries left: 5
E0213 12:56:45.172930   102 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0213 12:56:45.172963   102 HighLevelKafkaBase.cc:76] All kafka brokers down. Sleeping for: 756ms. 
I0213 12:56:45.180065   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: CONNECTED, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Connected (#6)
I0213 12:56:45.180099   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Sent ApiVersionRequest (v0, 56 bytes @ 0, CorrId 855)
I0213 12:56:45.180552   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RECV, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Received ApiVersionResponse (v0, 210 bytes, CorrId 855, rtt 0.43ms)
I0213 12:56:45.180596   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Sent FetchRequest (v4, 685 bytes @ 0, CorrId 856)
I0213 12:56:45.181512   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROKERFAIL, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)
I0213 12:56:45.401384   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: CONNECTED, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Connected (#340)
I0213 12:56:45.401425   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent ApiVersionRequest (v0, 56 bytes @ 0, CorrId 959)
I0213 12:56:45.401743   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RECV, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Received ApiVersionResponse (v0, 210 bytes, CorrId 959, rtt 0.30ms)
I0213 12:56:45.401799   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent FetchRequest (v4, 710 bytes @ 0, CorrId 960)
I0213 12:56:45.402194   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROKERFAIL, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)
I0213 12:56:45.402236   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RETRY, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Retrying FetchRequest (v4, 710 bytes, retry 1/2)
I0213 12:56:45.780860   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: CONNECTED, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Connected (#7)
I0213 12:56:45.780905   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Sent ApiVersionRequest (v0, 56 bytes @ 0, CorrId 857)
I0213 12:56:45.781422   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RECV, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Received ApiVersionResponse (v0, 210 bytes, CorrId 857, rtt 0.50ms)
I0213 12:56:45.781474   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Sent FetchRequest (v4, 685 bytes @ 0, CorrId 858)
I0213 12:56:45.782171   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROKERFAIL, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)
I0213 12:56:45.782219   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RETRY, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Retrying FetchRequest (v4, 685 bytes, retry 1/2)
I0213 12:56:45.915303   107 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 4, fac: COMMITFAIL, event: [thrd:main]: Offset commit (cgrp auto commit timer) failed for 60/60 partition(s): Local: Waiting for coordinator: logs.raw[0]@353506, logs.raw[1], logs.raw[2], logs.raw[3], logs.raw[4], logs.raw[5], logs.raw[6], logs.raw[7], logs.raw[8], logs.raw[9], logs.raw[10], logs.raw[11], logs.raw[12], logs.in..
E0213 12:56:45.929028   102 HighLevelKafkaBase.cc:80] Retries left: 4
E0213 12:56:45.929044   102 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0213 12:56:45.929080   102 HighLevelKafkaBase.cc:76] All kafka brokers down. Sleeping for: 391ms. 
I0213 12:56:46.068554   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: CONNECTED, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Connected (#341)
I0213 12:56:46.068594   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent ApiVersionRequest (v0, 56 bytes @ 0, CorrId 961)
I0213 12:56:46.068975   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RECV, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Received ApiVersionResponse (v0, 210 bytes, CorrId 961, rtt 0.36ms)
I0213 12:56:46.069018   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: SEND, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Sent FetchRequest (v4, 710 bytes @ 0, CorrId 962)
I0213 12:56:46.069526   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROKERFAIL, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)
I0213 12:56:46.069581   109 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: RETRY, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72543: Retrying FetchRequest (v4, 710 bytes, retry 2/2)
E0213 12:56:46.320147   102 HighLevelKafkaBase.cc:80] Retries left: 3

Provide broker log excerpts
No error message in broker
Critical issue

First , Pls check my client settings and let me know if anything should be modified...

Kafka Heap Settings:
echo $KAFKA_HEAP_OPTS
-Xms6g -Xmx20g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80

The text was updated successfully, but these errors were encountered:

edenhill · 2018-02-13T13:52:46Z

Huhm, it is weird that it doesn't say what response is malformed.
Can you try to reproduce this with debug=all and see if there is any additional information prior to the "Bad message format" log line?

It would also be valuable if you could reproduce this with latest librdkafka master

senthilec566 · 2018-02-13T17:56:57Z

@edenhill , Thanks ..

Attaching logs with debug=all..

debug_all.txt

I0213 17:38:30.464459   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROKERFAIL, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: failed: err: Local: Bad message format: (errno: Resource temporarily unavailable)
I0213 17:38:30.464473   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [8] 0+3 msgs
I0213 17:38:30.464485   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [26] 0+5 msgs
I0213 17:38:30.464493   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [2] 1+1 msgs
I0213 17:38:30.464501   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [20] 0+1 msgs
I0213 17:38:30.464509   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [14] 0+1 msgs
I0213 17:38:30.464517   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [4] 0+4 msgs
I0213 17:38:30.464525   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [22] 2+4 msgs
I0213 17:38:30.464522   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: STATE, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Broker changed state UP -> DOWN
I0213 17:38:30.464534   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [16] 1+2 msgs
I0213 17:38:30.464546   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BROADCAST, event: [thrd:0.0.0.24:9092/bootstrap]: Broadcasting state change
I0213 17:38:30.464546   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [10] 0+1 msgs
I0213 17:38:30.464557   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BUFQ, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Purging bufq with 2 buffers
I0213 17:38:30.464561   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [28] 1+1 msgs
I0213 17:38:30.464568   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: FETCH, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Fetch reply: Local: Bad message format
I0213 17:38:30.464572   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [18] 1+2 msgs
I0213 17:38:30.464579   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BACKOFF, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Fetch backoff for 500ms: Local: Bad message format
I0213 17:38:30.464581   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [12] 0+3 msgs
I0213 17:38:30.464597   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [6] 0+3 msgs
I0213 17:38:30.464607   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [24] 0+0 msgs
I0213 17:38:30.464599   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: BUFQ, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Updating 0 buffers on connection reset
I0213 17:38:30.464617   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.tranformed_rawlogs [0] 0+4 msgs
I0213 17:38:30.464623   110 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: METADATA, event: [thrd:0.0.0.24:9092/bootstrap]: Skipping metadata refresh of 2 topic(s): no usable brokers
I0213 17:38:30.464625   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.httpsink_data [8] 0+2 msgs
I0213 17:38:30.464609   107 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: REQERR, event: [thrd:main]: 0.0.0.24:9092/72542: HeartbeatRequest failed: Local: Bad message format: explicit actions 0x0
I0213 17:38:30.464640   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.httpsink_data [26] 2+4 msgs
I0213 17:38:30.464646   107 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: HEARTBEAT, event: [thrd:main]: Group "rawlogs_data_transform.rawlogs_logs.framework" heartbeat error response in state up (join state started, 60 partition(s) assigned): Local: Bad message format
I0213 17:38:30.464648   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.httpsink_data [20] 0+3 msgs
I0213 17:38:30.464658   107 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: CGRPJOINSTATE, event: [thrd:main]: Group "rawlogs_data_transform.rawlogs_logs.framework" changed join state started -> init (v4, state up)
I0213 17:38:30.464661   106 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: TOPPAR, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: rawlogs_logs.httpsink_data [2] 0+1 msgs
I0213 17:38:30.464665   107 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 7, fac: PAUSE, event: [thrd:main]: Library pausing 60 partition(s)

senthilec566 · 2018-02-13T18:05:50Z

adding last 10K log line...
debug_all_01.txt

edenhill · 2018-02-13T18:28:46Z

So what I think is happening here is that the broker (or the network) is disconnecting the TCP session.
Are you sure there's nothing in any of the broker logs?

edenhill · 2018-02-13T19:04:03Z

Is there anything special with your setup? Load balancers? Firewalls?

senthilec566 · 2018-02-14T07:17:27Z

Let me enable Kafka Brokers DEBUG logs and will keep you posted here.... Just wondering first few minutes the connectivity seems to be working in other words we were able to produce/consume ~50K messages ...

Is there anything special with your setup? Load balancers? Firewalls? - No.. Its my own test setup ...

Thanks @edenhill for checking this ...

senthilec566 · 2018-02-14T07:49:47Z

@edenhill Yes , You are absolutely correct...

Here is the DEBUG logs of Broker:

DEBUG Connection with /0.0.0.13 disconnected (org.apache.kafka.common.network.Selector)
java.io.IOException: Broken pipe
	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
	at org.apache.kafka.common.network.PlaintextTransportLayer.transferFrom(PlaintextTransportLayer.java:218)
	at org.apache.kafka.common.record.FileRecords.writeTo(FileRecords.java:263)
	at org.apache.kafka.common.requests.RecordsSend.writeTo(RecordsSend.java:57)
	at org.apache.kafka.common.network.MultiSend.writeTo(MultiSend.java:76)
	at org.apache.kafka.common.network.KafkaChannel.send(KafkaChannel.java:207)
	at org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:385)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
	at kafka.network.Processor.poll(SocketServer.scala:499)
	at kafka.network.Processor.run(SocketServer.scala:435)
	at java.lang.Thread.run(Thread.java:748)
[2018-02-14 07:24:10,795] DEBUG Connection with /0.0.0.13 disconnected (org.apache.kafka.common.network.Selector)
java.io.IOException: Broken pipe
	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
	at org.apache.kafka.common.network.PlaintextTransportLayer.transferFrom(PlaintextTransportLayer.java:218)
	at org.apache.kafka.common.record.FileRecords.writeTo(FileRecords.java:263)
	at org.apache.kafka.common.requests.RecordsSend.writeTo(RecordsSend.java:57)
	at org.apache.kafka.common.network.MultiSend.writeTo(MultiSend.java:76)
	at org.apache.kafka.common.network.KafkaChannel.send(KafkaChannel.java:207)
	at org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:385)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
	at kafka.network.Processor.poll(SocketServer.scala:499)
	at kafka.network.Processor.run(SocketServer.scala:435)
	at java.lang.Thread.run(Thread.java:748)
[2018-02-14 07:24:10,977] DEBUG Got ping response for sessionid: 0x3619317ae4c0000 after 0ms (org.apache.zookeeper.ClientCnxn)
[2018-02-14 07:24:11,223] DEBUG Connection with /0.0.0.13 disconnected (org.apache.kafka.common.network.Selector)
java.io.EOFException
	at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:87)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:75)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:203)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:167)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:379)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
	at kafka.network.Processor.poll(SocketServer.scala:499)

edenhill · 2018-02-14T08:01:08Z

Broken pipe means something other than the broker tore down the connection, in your case that must be the client.
So we're back to librdkafka again.
Could you get latest librdkafka master and change this line:
https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_buf.h#L162

if (log_decode_errors > 0) {

to:

if (1) {

and reproduce the issue with full debugging enabled?

senthilec566 · 2018-02-14T09:01:39Z

Thanks , I'll do it now .. Could you Pls provide steps to bundle from master ?

How to bundle like this https://github.com/edenhill/librdkafka/archive/v0.11.3.tar.gz ?? Sorry this is the first time im trying LibRdKafkaDriver :-) ..

edenhill · 2018-02-14T09:04:53Z

The easiest approach is to:

download https://github.com/edenhill/librdkafka/archive/master.tar.gz
unzip the file
cd into the directory
./configure --prefix=/usr
change the code as described above
make
sudo make install
Rebuild your program and rerun it

senthilec566 · 2018-02-14T09:11:12Z

Cool , Thanks .. I'm not directly running librdkafka program ... There is a another process which will use driver to run it .. So i need to give input file like this https://github.com/edenhill/librdkafka/archive/v0.11.3.tar.gz :-) ...

edenhill · 2018-02-14T09:13:22Z

You can create a tar ball of the change and provide that to your application perhaps.

senthilec566 · 2018-02-14T09:53:32Z

Cool , In few minutes , I'll attach logs ...

senthilec566 · 2018-02-14T10:16:46Z

debug_1682.txt

@edenhill , Here you go :

DEBUG Connection with /0.0.0.13 disconnected (org.apache.kafka.common.network.Selector)
java.io.EOFException
 at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:87)
 at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:75)
 at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:203)
 at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:167)
 at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:379)
 at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
 at kafka.network.Processor.poll(SocketServer.scala:499)
 at kafka.network.Processor.run(SocketServer.scala:435)
 at java.lang.Thread.run(Thread.java:748)
[2018-02-14 10:10:26,810] DEBUG Connection with /0.0.0.13 disconnected (org.apache.kafka.common.network.Selector)
java.io.EOFException
 at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:87)
 at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:75)
 at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:203)
 at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:167)
 at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:379)
 at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
 at kafka.network.Processor.poll(SocketServer.scala:499)
 at kafka.network.Processor.run(SocketServer.scala:435)
 at java.lang.Thread.run(Thread.java:748)



 DEBUG Connection with /0.0.0.13 disconnected (org.apache.kafka.common.network.Selector)
 java.io.IOException: Connection reset by peer
 	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
 	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
 	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
 	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
 	at org.apache.kafka.common.network.PlaintextTransportLayer.transferFrom(PlaintextTransportLayer.java:218)
 	at org.apache.kafka.common.record.FileRecords.writeTo(FileRecords.java:263)
 	at org.apache.kafka.common.requests.RecordsSend.writeTo(RecordsSend.java:57)
 	at org.apache.kafka.common.network.MultiSend.writeTo(MultiSend.java:76)
 	at org.apache.kafka.common.network.KafkaChannel.send(KafkaChannel.java:207)
 	at org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:178)
 	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:385)
 	at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
 	at kafka.network.Processor.poll(SocketServer.scala:499)
 	at kafka.network.Processor.run(SocketServer.scala:435)
 	at java.lang.Thread.run(Thread.java:748)
 [2018-02-14 10:11:56,359] DEBUG Connection with /0.0.0.13 disconnected (org.apache.kafka.common.network.Selector)
 java.io.IOException: Connection reset by peer
 	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
 	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
 	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
 	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
 	at org.apache.kafka.common.network.PlaintextTransportLayer.transferFrom(PlaintextTransportLayer.java:218)
 	at org.apache.kafka.common.record.FileRecords.writeTo(FileRecords.java:263)
 	at org.apache.kafka.common.requests.RecordsSend.writeTo(RecordsSend.java:57)
 	at org.apache.kafka.common.network.MultiSend.writeTo(MultiSend.java:76)
 	at org.apache.kafka.common.network.KafkaChannel.send(KafkaChannel.java:207)
 	at org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:178)
 	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:385)
 	at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
 	at kafka.network.Processor.poll(SocketServer.scala:499)

edenhill · 2018-02-14T10:27:56Z

Thanks!

And this is with a patched librdkafka?

senthilec566 · 2018-02-14T10:33:33Z

Yep ..
Steps:

Downloaded master https://github.com/edenhill/librdkafka/archive/master.tar.gz
https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_buf.h#L162 - Changed to if(1)
./configure
make
sudo make install
Re Run

debug_1682_01.txt

senthilec566 · 2018-02-14T10:36:55Z

@edenhill , Let me know if you are looking additional logs ...

edenhill · 2018-02-14T10:51:32Z

I can't find the printout that we patched, so it must be something else wrong.

It could be that the returned FetchResponse is too big, but this should be logged appropriately.
Can you try to set receive.message.max.bytes to 100000000?

senthilec566 · 2018-02-14T11:48:00Z

Client Settings:
message.max.bytes=67108864,socket.timeout.ms=10000,fetch.message.max.bytes=67108864,receive.message.max.bytes=100000000,max.partition.fetch.bytes=67108864,queue.buffering.max.ms=100,message.copy.max.bytes=67108864,broker.version.fallback=0.11.0.0,compression.codec=lz4,log.connection.close=false,debug=all

No luck after setting receive.message.max.bytes to 100000000.

sink_err.txt
debug_all_bad_01.txt

Pls refer the attached logs again and let me know if you are able to see the expected printout ...

edenhill · 2018-02-14T11:55:34Z

Try using default settings (remove your config) for all of the max.bytes, fetch.bytes, etc.
Also enable log.connection.close

senthilec566 · 2018-02-14T11:57:20Z

Sure , I'll run with recommended settings ...

senthilec566 · 2018-02-14T12:05:20Z

Still same issue ...

I0214 12:02:04.541733   111 HighLevelKafkaConsumer.cc:104] Overriding consumer cfg: debug=all
I0214 12:02:04.541743   111 HighLevelKafkaConsumer.cc:104] Overriding consumer cfg: log.connection.close=true

senthilec566 · 2018-02-14T12:06:27Z

Attaching debug_all logs...
default_setting.txt

senthilec566 · 2018-02-14T13:55:58Z

@edenhill , Let me know if you want to try with different settings ...

edenhill · 2018-02-14T13:57:25Z

I'm a bit stumped to be honest, I can't find a location in the code where ERR__BAD_MSG is propagated without a debug message being emitted

senthilec566 · 2018-02-15T08:38:26Z

@edenhill , Did you find anything ?...Let me know if you are looking for additional debug logs etc ...

also i could see consumer failed with same bad message error..

E0214 11:37:19.510848   104 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0214 11:37:19.511023   104 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0214 11:37:19.621539   104 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0214 11:37:19.621826   104 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0214 11:37:20.244236   104 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0214 11:37:20.244377   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.244458   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.244537   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.244616   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.244689   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.244774   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.244845   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.244918   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.244989   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.245059   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.245131   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.245200   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.245271   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.245342   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.245411   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305344   104 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0214 11:37:20.305361   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305366   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305368   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305374   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305433   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305438   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305440   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305444   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305480   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305521   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305560   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305598   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305635   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.305687   104 HighLevelKafkaConsumer.hpp:85] Consume failed. Uknown reason: Local: Bad message format
E0214 11:37:20.841215   104 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format

edenhill · 2018-02-15T14:14:13Z

I want to add some extra debugging to track this down, would you be fine with testing a special branch?

senthilec566 · 2018-02-15T14:16:20Z

@edenhill - Yep , Im fine with testing a special branch ....

senthilec566 · 2018-02-16T20:49:23Z

@edenhill , Pls share me branch details if you are done with additional changes ...

senthilec566 · 2018-02-19T12:45:23Z

@edenhill , Ping ...

edenhill · 2018-02-19T13:11:01Z

Please try the badmsg branch and reproduce the issue with debug=all and provide the full log

senthilec566 · 2018-02-19T14:59:32Z

Thanks , I'll test with this branch ...

senthilec566 · 2018-02-20T16:04:07Z

My current setup have some problem.. Will test this tomorrow...

senthilec566 · 2018-02-24T07:36:47Z

Sorry for the delay in testing ...

without enabling debug_all i could see below:

E0224 07:35:26.675529   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0224 07:35:26.675562   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:27.175838   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:27.194568   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:27.316421   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:27.316484   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:27.797909   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
I0224 07:35:27.798945   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:27.814865   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0224 07:35:27.814899   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:28.367913   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:28.369513   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:28.385982   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:28.391106   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:28.977069   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:28.991883   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:29.061780   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:29.078500   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:29.723044   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
I0224 07:35:29.757041   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:29.769698   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0224 07:35:29.799638   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:30.202334   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:30.271234   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:30.299283   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555115 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:30.369093   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:30.896641   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
I0224 07:35:30.908069   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:31.163128   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0224 07:35:31.187520   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:31.435796   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555153 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:31.522012   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:31.538739   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:31.593611   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:31.834911   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:31.881884   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:32.028527   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555115 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:32.037976   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:32.542722   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:32.623982   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:32.720974   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:32.763566   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:33.151429   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:33.175607   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:33.271216   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:33.301878   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:33.592412   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:33.601809   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:35:33.901995   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/72542: Invalid response size 33555148 (0..33554432): increase receive.message.max.bytes: Local: Bad message format
E0224 07:35:33.917691   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format```

senthilec566 · 2018-02-24T07:52:55Z

Tried this : log.connection.close=true,receive.message.max.bytes=100000000


I0224 07:51:07.706616   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000350 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
I0224 07:51:07.706655   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000443 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:07.706670   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
E0224 07:51:07.706720   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:07.807610   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:07.807677   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:07.807847   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:07.807929   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:08.355258   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:08.355307   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:08.381028   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:08.381098   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:08.944447   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:08.944497   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:09.048737   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:09.048835   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:09.521883   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:09.521945   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:09.686278   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:09.686352   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:10.149443   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:10.149525   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:10.405485   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:10.405609   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:10.782729   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:10.782814   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:11.050858   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:11.050920   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:11.471884   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:11.471935   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:11.676967   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:11.677019   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:12.032122   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:12.032166   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:12.333218   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:12.333283   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:12.555399   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:12.555450   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:12.872324   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000721 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:12.872380   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:13.197623   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000683 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:13.197686   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:13.268337   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000683 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:13.268381   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:13.815845   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:13.815899   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:13.817433   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:13.817486   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:14.375562   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:14.375619   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:14.426090   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:14.426138   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:15.056902   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:15.056958   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:15.165272   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:15.165359   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:15.429101   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:15.429168   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:15.864423   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:15.864523   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:16.093228   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:16.093355   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:16.533704   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:16.533777   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:16.712553   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:16.712687   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:17.194948   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:17.195066   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:17.459913   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:17.460016   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:17.937243   113 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.24:9092/bootstrap]: 0.0.0.24:9092/72542: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:17.937306   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format
I0224 07:51:18.193346   111 HighLevelKafkaBase.cc:95] Librdkafka log: severity: 3, fac: INVALID, event: [thrd:0.0.0.30:9092/bootstrap]: 0.0.0.30:9092/72543: Invalid response size 100000716 (0..100000000): increase receive.message.max.bytes: Local: Bad message format
E0224 07:51:18.193440   103 HighLevelKafkaBase.cc:87] Librdkafka error: Local: Bad message format

senthilec566 · 2018-02-24T07:57:30Z

I've increased size to
log.connection.close=true,receive.message.max.bytes=1000000000
Rerunning test case again..

senthilec566 · 2018-02-24T10:56:47Z

Consumer is running fine with load 5Mbps ... I'll increase the load and will keep you posted ...

edenhill · 2018-03-06T09:45:36Z

Any updates?

senthilec566 · 2018-03-10T11:01:02Z

After increasing receive.max.message.bytes I'm not finding bad error message.

total no of partitions in topic : 30 ...

senthilec566 · 2018-03-12T03:19:17Z

@edenhill . I ran a test for three days continuously with load ~4Mbps and i dont see the this bad message error ...

What is your recommendation of LibRdKafka Settings?

Total Number of Partitions in a Topic : 30. Expected traffic ~30Mbps ...
Avg Message Size : 2KB
Max Message Size : 10KB
Broker Settings:

socket.send.buffer.bytes = 102400
num.partitions = 30
socket.receive.buffer.bytes = 102400
message.max.bytes = 67108864
auto.create.topics.enable = true
replica.fetch.max.bytes = 67108864
socket.request.max.bytes = 104857600
default.replication.factor = 2
offsets.topic.replication.factor = 2
compression.codec = 3

I'll increase the load and let you know the status here..

Current Driver Settings:

log.connection.close=true,receive.message.max.bytes=1000000000,queue.buffering.max.ms=10,broker.version.fallback=0.11.0.0,compression.codec=lz4

edenhill · 2018-03-12T09:42:01Z

That's a moderate traffic load and I think you can use default values for pretty much everything.

senthilec566 · 2018-03-15T02:53:57Z

@edenhill , I'm frequently getting message timeout error @7Mbps load..
Librdkafka error: Local: Message timed out

No errors found in broker logs too..

Driver Settings:
log.connection.close=true,receive.message.max.bytes=1000000000,queue.buffering.max.ms=10,broker.version.fallback=0.11.0.0,compression.codec=lz4

edenhill · 2018-03-15T07:09:33Z

Enable debug=protocol and look at the response times (rtt) of requests, do they increase over time until finally timing out?

You could try to set queue.buffering.backpressure.threshold=1 to improve batching, and perhaps increase queue.buffering.max.ms to 100.

edenhill · 2018-03-20T08:38:22Z

dup of #1616

edenhill added a commit that referenced this issue Feb 19, 2018

Extra debugging to track down #1682

3b9111d

edenhill added a commit that referenced this issue Mar 13, 2018

Extra debugging to track down #1682

9223b09

edenhill added the duplicate label Mar 20, 2018

edenhill closed this as completed Mar 20, 2018

BROKERFAIL : failed: err: Local: Bad message format: (errno: Resource temporarily unavailable) #1682

BROKERFAIL : failed: err: Local: Bad message format: (errno: Resource temporarily unavailable) #1682

Comments

senthilec566 commented Feb 13, 2018 • edited Loading

Description

How to reproduce

Checklist

edenhill commented Feb 13, 2018

senthilec566 commented Feb 13, 2018 • edited Loading

senthilec566 commented Feb 13, 2018

edenhill commented Feb 13, 2018

edenhill commented Feb 13, 2018

senthilec566 commented Feb 14, 2018

senthilec566 commented Feb 14, 2018

edenhill commented Feb 14, 2018

senthilec566 commented Feb 14, 2018 • edited Loading

edenhill commented Feb 14, 2018 • edited Loading

senthilec566 commented Feb 14, 2018

edenhill commented Feb 14, 2018

senthilec566 commented Feb 14, 2018

senthilec566 commented Feb 14, 2018

edenhill commented Feb 14, 2018

senthilec566 commented Feb 14, 2018

senthilec566 commented Feb 14, 2018

edenhill commented Feb 14, 2018

senthilec566 commented Feb 14, 2018 • edited Loading

edenhill commented Feb 14, 2018

senthilec566 commented Feb 14, 2018

senthilec566 commented Feb 14, 2018 • edited Loading

senthilec566 commented Feb 14, 2018

senthilec566 commented Feb 14, 2018 • edited Loading

edenhill commented Feb 14, 2018

senthilec566 commented Feb 15, 2018 • edited Loading

edenhill commented Feb 15, 2018

senthilec566 commented Feb 15, 2018

senthilec566 commented Feb 16, 2018 • edited Loading

senthilec566 commented Feb 19, 2018

edenhill commented Feb 19, 2018

senthilec566 commented Feb 19, 2018

senthilec566 commented Feb 20, 2018

senthilec566 commented Feb 24, 2018 • edited Loading

senthilec566 commented Feb 24, 2018

senthilec566 commented Feb 24, 2018

senthilec566 commented Feb 24, 2018

edenhill commented Mar 6, 2018

senthilec566 commented Mar 10, 2018

senthilec566 commented Mar 12, 2018 • edited Loading

edenhill commented Mar 12, 2018

senthilec566 commented Mar 15, 2018 • edited Loading

edenhill commented Mar 15, 2018 • edited Loading

edenhill commented Mar 20, 2018

senthilec566 commented Feb 13, 2018 •

edited

Loading

senthilec566 commented Feb 13, 2018 •

edited

Loading

senthilec566 commented Feb 14, 2018 •

edited

Loading

edenhill commented Feb 14, 2018 •

edited

Loading

senthilec566 commented Feb 14, 2018 •

edited

Loading

senthilec566 commented Feb 14, 2018 •

edited

Loading

senthilec566 commented Feb 14, 2018 •

edited

Loading

senthilec566 commented Feb 15, 2018 •

edited

Loading

senthilec566 commented Feb 16, 2018 •

edited

Loading

senthilec566 commented Feb 24, 2018 •

edited

Loading

senthilec566 commented Mar 12, 2018 •

edited

Loading

senthilec566 commented Mar 15, 2018 •

edited

Loading

edenhill commented Mar 15, 2018 •

edited

Loading