Does the library support broker recover back #27

xiaojun789 · 2013-11-21T06:55:04Z

Hi,
In my case, when the kafka cluster is shutdown, and the producer use the librdkafka still try to send the log to kafka, it will failed.
but When I start the Kafka cluster, the producer also can't send the log to kafka, and I found the socket in producer is "CLOSE_WAIT" status

[irteam@xseed021.kdev logs]$ netstat -an|grep 9093
tcp        1      0 10.96.250.211:43934         10.99.116.53:9093           CLOSE_WAIT  
tcp        1      0 10.96.250.211:50729         10.99.116.54:9093           CLOSE_WAIT  
tcp        1      0 10.96.250.211:45626         10.99.116.55:9093           CLOSE_WAIT  
tcp        1      0 10.96.250.211:45625         10.99.116.55:9093           CLOSE_WAIT  
tcp        1      0 10.96.250.211:43951         10.99.116.53:9093           CLOSE_WAIT

Does the librdlibary has the feature to support checking broker alive? Thanks~!

The text was updated successfully, but these errors were encountered:

edenhill · 2013-11-21T12:57:50Z

Hi,

this sounds weird.
CLOSE_WAIT means the broker closed the connection but the local application has not closed the socket yet.
rdkafka constantly polls its broker sockets for hangups and should thus pick up on the hangup quite quickly and close the socket accordingly.
I've not been able to reproduce this issue.

Could you try to enable debugging in rdkafka while reproducing this issue?
Call rd_kafka_conf_set(conf, "debug", "all", ...); prior to creating the rd_kafka_t handle.

xiaojun789 · 2013-11-27T07:47:52Z

HI,
It's weired, I can't reproduce again, also I found the library can check the server recover.

but, when I don't test, I found a log is very strange

RDKAFKA-5-PARTCNT: rdkafka#producer-0: Topic kafka-perf partition count changed from 1 to 0

my test step:

start the kafka service
produce send the log to kafka
stop the kafka service
start the service
and the log will happened. and then, I can't send any data to kafka, but I saw the connection is established.

edenhill · 2013-11-27T08:42:54Z

Could you try to enable debugging in librdkafka?
If you have written your own program using librdkafka you will need to enable debugging through the
configuration property interface:
Call rd_kafka_conf_set(conf, "debug", "all", ...); prior to creating the rd_kafka_t handle.

If you are using rdkafka_example or rdkafka_performance all you need to do is provide the "-d all" command-line switch.

After broker restart it might take up to topic.metadata.refresh.interval.ms (default 10s) before the partitions are available to rdkafka.

xiaojun789 · 2013-11-27T10:02:10Z

I have enabled the debuging as you mentioned.
normally, when I stop the kafka server
the log will as folloiwng

[2013-11-27 17:45:28.983] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-CONNECT: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: broker in state DOWN connecting

[2013-11-27 17:45:28.984] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Broker changed state DOWN -> CONNECTING

[2013-11-27 17:45:28.984] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-CONNECT: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: couldn't connect to ipv4#10.34.130.76:9092: Connection refused

[2013-11-27 17:45:28.984] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BROKERFAIL: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: failed: err: Local: Broker transport failure: (errno: Connection refused)

[2013-11-27 17:45:28.984] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Broker changed state CONNECTING -> DOWN

[2013-11-27 17:45:28.984] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-3-FAIL: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: nelo76.nhnchengdu.com:9092/0: Failed to connect to broker at nelo76.nhnchengdu.com:9092: Connection
refused

[2013-11-27 17:45:28.984] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Purging bufq

[2013-11-27 17:45:28.984] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Purging bufq

[2013-11-27 17:45:28.984] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-CONNECT: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: broker in state DOWN connecting

[2013-11-27 17:45:28.984] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Broker changed state DOWN -> CONNECTING

[2013-11-27 17:45:28.984] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-CONNECT: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: couldn't connect to ipv4#10.34.130.76:9092: Connection refused

[2013-11-27 17:45:28.984] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BROKERFAIL: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: failed: err: Local: Broker transport failure: (errno: Connection refused)

[2013-11-27 17:45:28.984] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Broker changed state CONNECTING -> DOWN

[2013-11-27 17:45:28.984] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-3-FAIL: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: nelo76.nhnchengdu.com:9092/bootstrap: Failed to connect to broker at nelo76.nhnchengdu.com:9
092: Connection refused

[2013-11-27 17:45:28.984] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Purging bufq

[2013-11-27 17:45:28.984] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Purging bufq

[2013-11-27 17:45:28.986] < 6367-0x5576e940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 5 ops

[2013-11-27 17:45:28.986] < 6367-0x5576e940> [INFO][kafkaSink.cpp:558] The broker is dead
[2013-11-27 17:45:28.986] < 6367-0x5576e940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/bootstrap: Receive failed: Receive error: Connection reset by peer
[2013-11-27 17:45:28.986] < 6367-0x5576e940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/0: Receive failed: Disconnected
[2013-11-27 17:45:28.986] < 6367-0x5576e940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/bootstrap: Connection closed 
[2013-11-27 17:45:28.986] < 6367-0x5576e940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/0: Failed to connect to broker at nelo76.nhnchengdu.com:9092: Connection refused
[2013-11-27 17:45:28.986] < 6367-0x5576e940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/bootstrap: Failed to connect to broker at nelo76.nhnchengdu.com:9092: Connection refused
[2013-11-27 17:45:29.648] < 6367-0x57571940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

after I start the server again the log is following

[2013-11-27 17:45:39.878] < 6367-0x5616f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PART: rdkafka#producer-0: kafka-perf has no partitions

[2013-11-27 17:45:39.878] < 6367-0x5616f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PART: rdkafka#producer-0: kafka-perf has no partitions

[2013-11-27 17:45:39.878] < 6367-0x5616f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PART: rdkafka#producer-0: kafka-perf has no partitions

[2013-11-27 17:45:39.878] < 6367-0x5616f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PART: rdkafka#producer-0: kafka-perf has no partitions

[2013-11-27 17:45:39.878] < 6367-0x5616f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PART: rdkafka#producer-0: kafka-perf has no partitions

[2013-11-27 17:45:39.878] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: ===== Received metadata from nelo76.nhnchengdu.com:9092/bootstrap =====

[2013-11-27 17:45:39.878] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: 1 brokers

[2013-11-27 17:45:39.878] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap:   Broker #0/1: nelo76.nhnchengdu.com:9092 NodeId 0

[2013-11-27 17:45:39.879] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: Topic kafka-perf partition count changed from 0 to 1

[2013-11-27 17:45:39.879] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap:   Topic #0/1: kafka-perf partition 0 Leader 0

[2013-11-27 17:45:39.879] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BRKDELGT: rdkafka#producer-0: Broker nelo76.nhnchengdu.com:9092/0 is now leader for topic kafka-perf [0] with 0 messages (0 bytes) queued

[2013-11-27 17:45:39.879] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: Partitioning 33190 unassigned messages in topic kafka-perf to 1 partitions

[2013-11-27 17:45:39.889] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-UAS: rdkafka#producer-0: 33190/33190 messages were partitioned

[2013-11-27 17:45:39.889] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: ===== Received metadata from nelo76.nhnchengdu.com:9092/bootstrap =====

[2013-11-27 17:45:39.889] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: 1 brokers

[2013-11-27 17:45:39.889] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap:   Broker #0/1: nelo76.nhnchengdu.com:9092 NodeId 0

[2013-11-27 17:45:39.889] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: No change in partition count for topic kafka-perf

[2013-11-27 17:45:39.889] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap:   Topic #0/1: kafka-perf partition 0 Leader 0

[2013-11-27 17:45:39.889] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPICUPD: rdkafka#producer-0: No leader change for topic kafka-perf [0] with leader 0

[2013-11-27 17:45:39.890] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: Partitioning 0 unassigned messages in topic kafka-perf to 1 partitions

[2013-11-27 17:45:39.890] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-UAS: rdkafka#producer-0: 0/0 messages were partitioned

but after do this serveral time, there is no debug message come from librdkakfa

[2013-11-27 17:49:39.479] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) not delivered

[2013-11-27 17:49:39.479] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Purging bufq

[2013-11-27 17:49:39.479] < 6367-0x5d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) encountered error: Local: Broker transport failure

[2013-11-27 17:49:39.479] < 6367-0x4a35c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Purging bufq

[2013-11-27 17:49:39.483] < 6367-0x50766940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-27 17:49:39.483] < 6367-0x50766940> [INFO][kafkaSink.cpp:558] The broker is dead
[2013-11-27 17:49:39.483] < 6367-0x50766940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/0: Receive failed: Receive error: Connection reset by peer
[2013-11-27 17:49:39.483] < 6367-0x50766940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/bootstrap: Receive failed: Disconnected
[2013-11-27 17:49:43.757] < 6364-0x4214f940> [INFO][processMasterSignal.cpp:145] Monitor thread of master process(6364) is working.

edenhill · 2013-11-27T10:15:49Z

Are there no kafka debug messages between [2013-11-27 17:45:39.890] and [2013-11-27 17:49:39.479] ?

xiaojun789 · 2013-11-28T01:33:47Z

this morning when I stop the kafka and start again, there will be no more log form librdkafka. following is the whole log message

[2013-11-28 09:22:10.460] < 6275-0x42f6a940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BROKER: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Added new broker with NodeId -1

[2013-11-28 09:22:10.460] < 6275-0x42f6a940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPIC: rdkafka#producer-0: New local topic: kafka-perf

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BRKMAIN: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Enter main broker thread

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-CONNECT: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: broker in state DOWN connecting

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Broker changed state DOWN -> CONNECTING

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-CONNECTED: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: connected to nelo76.nhnchengdu.com:9092

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Broker changed state CONNECTING -> UP

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Request metadata for all topics: connected

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Requesting metadata for all topics

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Sent 1 bufs

[2013-11-28 09:22:10.460] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Sent 0 bufs

[2013-11-28 09:22:10.460] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: ===== Received metadata from nelo76.nhnchengdu.com:9092/bootstrap =====

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: 1 brokers

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap:   Broker #0/1: nelo76.nhnchengdu.com:9092 NodeId 0

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BROKER: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Added new broker with NodeId 0

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: Topic kafka-perf partition count changed from 0 to 1

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap:   Topic #0/1: kafka-perf partition 0 Leader 0

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BRKDELGT: rdkafka#producer-0: Broker nelo76.nhnchengdu.com:9092/0 is now leader for topic kafka-perf [0] with 0 messages (0 bytes) queued

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: Partitioning 0 unassigned messages in topic kafka-perf to 1 partitions

[2013-11-28 09:22:10.463] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-UAS: rdkafka#producer-0: 0/0 messages were partitioned

[2013-11-28 09:22:10.463] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BRKMAIN: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Enter main broker thread

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-CONNECT: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: broker in state DOWN connecting

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Broker changed state DOWN -> CONNECTING

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-CONNECTED: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: connected to nelo76.nhnchengdu.com:9092

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Broker changed state CONNECTING -> UP

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Request metadata for all topics: connected

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Requesting metadata for all topics

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 1 bufs

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 0 bufs

[2013-11-28 09:22:10.464] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:10.464] < 6275-0x42f6a940> [INFO][neloSource.cpp:117] END: open source(thrift) is successful
[2013-11-28 09:22:10.465] < 6275-0x42f6a940> [INFO][processChild.cpp:468] Initialize child process's resources is successful.
[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: ===== Received metadata from nelo76.nhnchengdu.com:9092/0 =====

[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: 1 brokers

[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0:   Broker #0/1: nelo76.nhnchengdu.com:9092 NodeId 0

[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: No change in partition count for topic kafka-perf

[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0:   Topic #0/1: kafka-perf partition 0 Leader 0

[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPICUPD: rdkafka#producer-0: No leader change for topic kafka-perf [0] with leader 0

[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: Partitioning 0 unassigned messages in topic kafka-perf to 1 partitions

[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-UAS: rdkafka#producer-0: 0/0 messages were partitioned

[2013-11-28 09:22:10.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:11.461] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:11.465] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:12.463] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:12.466] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:13.465] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:13.467] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:14.471] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:14.482] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:15.473] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:15.485] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:16.474] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:16.488] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:17.474] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:17.491] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:18.475] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:18.492] < 6275-0x4b177940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.474] < 6275-0x4d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.476] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 2080+0 msgs

[2013-11-28 09:22:19.477] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:19.477] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:19.477] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:19.478] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:19.479] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:19.479] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:19.480] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:19.480] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:19.480] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 64 messages (11456 bytes)

[2013-11-28 09:22:19.480] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+0 msgs

[2013-11-28 09:22:19.481] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:19.482] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:19.512] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:19.512] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 3 bufs

[2013-11-28 09:22:19.512] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 0 bufs

[2013-11-28 09:22:19.512] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 116+20 msgs

[2013-11-28 09:22:19.513] < 6275-0x52983940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.523] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:19.524] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 60+136 msgs

[2013-11-28 09:22:19.528] < 6275-0x54786940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.532] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:19.532] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+196 msgs

[2013-11-28 09:22:19.533] < 6275-0x55187940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.540] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:19.540] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 40+216 msgs

[2013-11-28 09:22:19.542] < 6275-0x56589940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.548] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:19.549] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 40+256 msgs

[2013-11-28 09:22:19.552] < 6275-0x5798b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.557] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:19.557] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+296 msgs

[2013-11-28 09:22:19.558] < 6275-0x4bb78940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.565] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:19.565] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 40+316 msgs

[2013-11-28 09:22:19.569] < 6275-0x4cf7a940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.572] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:19.572] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+356 msgs

[2013-11-28 09:22:19.574] < 6275-0x4d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:19.575] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 64 message(s) delivered

[2013-11-28 09:22:19.575] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+376 msgs

[2013-11-28 09:22:19.579] < 6275-0x4e37c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.470] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Request metadata for all topics: periodic refresh

[2013-11-28 09:22:20.471] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Requesting metadata for all topics

[2013-11-28 09:22:20.471] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Sent 1 bufs

[2013-11-28 09:22:20.471] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Sent 0 bufs

[2013-11-28 09:22:20.473] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: ===== Received metadata from nelo76.nhnchengdu.com:9092/bootstrap =====

[2013-11-28 09:22:20.473] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: 1 brokers

[2013-11-28 09:22:20.473] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap:   Broker #0/1: nelo76.nhnchengdu.com:9092 NodeId 0

[2013-11-28 09:22:20.473] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: No change in partition count for topic kafka-perf

[2013-11-28 09:22:20.473] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap:   Topic #0/1: kafka-perf partition 0 Leader 0

[2013-11-28 09:22:20.474] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPICUPD: rdkafka#producer-0: No leader change for topic kafka-perf [0] with leader 0

[2013-11-28 09:22:20.474] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: Partitioning 0 unassigned messages in topic kafka-perf to 1 partitions

[2013-11-28 09:22:20.474] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-UAS: rdkafka#producer-0: 0/0 messages were partitioned

[2013-11-28 09:22:20.477] < 6275-0x51581940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.575] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 3671+396 msgs

[2013-11-28 09:22:20.576] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.576] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.576] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.576] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.576] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.577] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.577] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.577] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.577] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.577] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.578] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.578] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.578] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.578] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.579] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.579] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:20.579] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 35 messages (6265 bytes)

[2013-11-28 09:22:20.579] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+0 msgs

[2013-11-28 09:22:20.579] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Request metadata for all topics: periodic refresh

[2013-11-28 09:22:20.579] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Requesting metadata for all topics

[2013-11-28 09:22:20.579] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:20.585] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.585] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+20 msgs

[2013-11-28 09:22:20.586] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+40 msgs

[2013-11-28 09:22:20.586] < 6275-0x51f82940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: ===== Received metadata from nelo76.nhnchengdu.com:9092/0 =====

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: 1 brokers

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0:   Broker #0/1: nelo76.nhnchengdu.com:9092 NodeId 0

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: No change in partition count for topic kafka-perf

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-METADATA: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0:   Topic #0/1: kafka-perf partition 0 Leader 0

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPICUPD: rdkafka#producer-0: No leader change for topic kafka-perf [0] with leader 0

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PARTCNT: rdkafka#producer-0: Partitioning 0 unassigned messages in topic kafka-perf to 1 partitions

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-UAS: rdkafka#producer-0: 0/0 messages were partitioned

[2013-11-28 09:22:20.588] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+40 msgs

[2013-11-28 09:22:20.595] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.595] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+60 msgs

[2013-11-28 09:22:20.596] < 6275-0x53384940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.601] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.601] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+80 msgs

[2013-11-28 09:22:20.601] < 6275-0x53d85940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.607] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.607] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 40+100 msgs

[2013-11-28 09:22:20.612] < 6275-0x55187940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.619] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.620] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 40+140 msgs

[2013-11-28 09:22:20.620] < 6275-0x56589940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.627] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.628] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+180 msgs

[2013-11-28 09:22:20.628] < 6275-0x56f8a940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.629] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.630] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 3 bufs

[2013-11-28 09:22:20.630] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 0 bufs

[2013-11-28 09:22:20.630] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+200 msgs

[2013-11-28 09:22:20.633] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.633] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+220 msgs

[2013-11-28 09:22:20.637] < 6275-0x5798b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:20.639] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.639] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+220 msgs

[2013-11-28 09:22:20.639] < 6275-0x4bb78940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.647] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.647] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 40+240 msgs

[2013-11-28 09:22:20.650] < 6275-0x4cf7a940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.650] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.650] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 1+280 msgs

[2013-11-28 09:22:20.652] < 6275-0x4cf7a940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:20.655] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.655] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 39+281 msgs

[2013-11-28 09:22:20.658] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.658] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+320 msgs

[2013-11-28 09:22:20.659] < 6275-0x4e37c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:20.661] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.661] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+320 msgs

[2013-11-28 09:22:20.664] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.664] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+340 msgs

[2013-11-28 09:22:20.665] < 6275-0x4ed7d940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:20.668] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:20.668] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+340 msgs

[2013-11-28 09:22:20.670] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 35 message(s) delivered

[2013-11-28 09:22:20.670] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+360 msgs

[2013-11-28 09:22:20.671] < 6275-0x4f77e940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:21.478] < 6275-0x5017f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:21.669] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 3135+360 msgs

[2013-11-28 09:22:21.670] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.670] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.670] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.670] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.671] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.671] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.671] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.671] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.671] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.671] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.672] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.672] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.672] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:21.672] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 219 messages (39201 bytes)

[2013-11-28 09:22:21.672] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+0 msgs

[2013-11-28 09:22:21.673] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:21.676] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.676] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:21.677] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:21.678] < 6275-0x4e37c940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:21.680] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.680] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+20 msgs

[2013-11-28 09:22:21.683] < 6275-0x4ed7d940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:21.684] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.684] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+40 msgs

[2013-11-28 09:22:21.688] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.688] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+60 msgs

[2013-11-28 09:22:21.691] < 6275-0x4f77e940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:21.693] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.693] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 2 bufs

[2013-11-28 09:22:21.693] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 0 bufs

[2013-11-28 09:22:21.693] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+60 msgs

[2013-11-28 09:22:21.695] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.696] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+80 msgs

[2013-11-28 09:22:21.697] < 6275-0x5017f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:21.699] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.699] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+80 msgs

[2013-11-28 09:22:21.703] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.703] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+100 msgs

[2013-11-28 09:22:21.704] < 6275-0x50b80940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:21.707] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.707] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+100 msgs

[2013-11-28 09:22:21.709] < 6275-0x51581940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:21.711] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.711] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+120 msgs

[2013-11-28 09:22:21.715] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.716] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+140 msgs

[2013-11-28 09:22:21.716] < 6275-0x51f82940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:21.719] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.719] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+140 msgs

[2013-11-28 09:22:21.721] < 6275-0x52983940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:21.725] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:21.726] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+160 msgs

[2013-11-28 09:22:21.727] < 6275-0x53384940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:21.728] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 219 message(s) delivered

[2013-11-28 09:22:21.728] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+180 msgs

[2013-11-28 09:22:21.731] < 6275-0x53d85940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:22.485] < 6275-0x54786940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:22.728] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 3033+200 msgs

[2013-11-28 09:22:22.728] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.728] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.728] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.729] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.729] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.729] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.729] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.730] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.730] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.730] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.731] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.731] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:22.731] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 209 messages (37411 bytes)

[2013-11-28 09:22:22.731] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:22.732] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:22.736] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.736] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+0 msgs

[2013-11-28 09:22:22.737] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:22.743] < 6275-0x4f77e940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:22.747] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.747] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+20 msgs

[2013-11-28 09:22:22.754] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.754] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+40 msgs

[2013-11-28 09:22:22.756] < 6275-0x50b80940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:22.758] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.758] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 40+40 msgs

[2013-11-28 09:22:22.761] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.761] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+80 msgs

[2013-11-28 09:22:22.761] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 1 bufs

[2013-11-28 09:22:22.761] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 0 bufs

[2013-11-28 09:22:22.761] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+80 msgs

[2013-11-28 09:22:22.764] < 6275-0x51581940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:22.764] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered
···

xiaojun789 · 2013-11-28T01:34:07Z

continue


[2013-11-28 09:22:22.764] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 9+80 msgs

[2013-11-28 09:22:22.764] < 6275-0x51581940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:22.772] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.772] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 11+89 msgs

[2013-11-28 09:22:22.774] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.774] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+100 msgs

[2013-11-28 09:22:22.775] < 6275-0x51f82940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:22.779] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.779] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 5+100 msgs

[2013-11-28 09:22:22.784] < 6275-0x51f82940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:22.796] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.796] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 55+105 msgs

[2013-11-28 09:22:22.800] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.800] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+160 msgs

[2013-11-28 09:22:22.803] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:22.803] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+160 msgs

[2013-11-28 09:22:22.806] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 209 message(s) delivered

[2013-11-28 09:22:22.806] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+160 msgs

[2013-11-28 09:22:22.808] < 6275-0x53d85940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 4 ops

[2013-11-28 09:22:23.484] < 6275-0x4c579940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:23.807] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 2660+160 msgs

[2013-11-28 09:22:23.807] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.807] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.808] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.808] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.808] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.808] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.809] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.809] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.809] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.809] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.810] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:23.810] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 48 messages (8592 bytes)

[2013-11-28 09:22:23.810] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:23.810] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+0 msgs

[2013-11-28 09:22:23.813] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.814] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:23.814] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:23.815] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.815] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:23.817] < 6275-0x50b80940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:23.818] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.818] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 2 bufs

[2013-11-28 09:22:23.818] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 0 bufs

[2013-11-28 09:22:23.818] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+20 msgs

[2013-11-28 09:22:23.821] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.821] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+40 msgs

[2013-11-28 09:22:23.823] < 6275-0x5017f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:23.823] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.823] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 1+40 msgs

[2013-11-28 09:22:23.824] < 6275-0x5017f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:23.826] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.826] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 19+41 msgs

[2013-11-28 09:22:23.830] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.830] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+60 msgs

[2013-11-28 09:22:23.831] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.831] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+60 msgs

[2013-11-28 09:22:23.831] < 6275-0x51581940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 3 ops

[2013-11-28 09:22:23.835] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.835] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+60 msgs

[2013-11-28 09:22:23.838] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.838] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+80 msgs

[2013-11-28 09:22:23.839] < 6275-0x51f82940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:23.840] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:23.840] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+80 msgs

[2013-11-28 09:22:23.842] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 48 message(s) delivered

[2013-11-28 09:22:23.842] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+100 msgs

[2013-11-28 09:22:23.844] < 6275-0x52983940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:24.490] < 6275-0x4d97b940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:24.842] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 3062+100 msgs

[2013-11-28 09:22:24.843] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.843] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.843] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.843] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.843] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.844] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.844] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.844] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.844] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.844] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.845] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.845] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:24.845] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 138 messages (24702 bytes)

[2013-11-28 09:22:24.845] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 11+0 msgs

[2013-11-28 09:22:24.846] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 8+12 msgs

[2013-11-28 09:22:24.847] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.848] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:24.849] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.850] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+20 msgs

[2013-11-28 09:22:24.851] < 6275-0x4f77e940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:24.852] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.852] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+20 msgs

[2013-11-28 09:22:24.853] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.854] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 2 bufs

[2013-11-28 09:22:24.854] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-SEND: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Sent 0 bufs

[2013-11-28 09:22:24.854] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+40 msgs

[2013-11-28 09:22:24.856] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.856] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+40 msgs

[2013-11-28 09:22:24.857] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.857] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+40 msgs

[2013-11-28 09:22:24.858] < 6275-0x50b80940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 4 ops

[2013-11-28 09:22:24.861] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.861] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+40 msgs

[2013-11-28 09:22:24.862] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.862] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+60 msgs

[2013-11-28 09:22:24.864] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.864] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+60 msgs

[2013-11-28 09:22:24.865] < 6275-0x5017f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 3 ops

[2013-11-28 09:22:24.866] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.866] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+60 msgs

[2013-11-28 09:22:24.868] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.868] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+80 msgs

[2013-11-28 09:22:24.870] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) delivered

[2013-11-28 09:22:24.870] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+80 msgs

[2013-11-28 09:22:24.871] < 6275-0x51581940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 3 ops

[2013-11-28 09:22:24.872] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 138 message(s) delivered

[2013-11-28 09:22:24.872] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 20+80 msgs

[2013-11-28 09:22:24.876] < 6275-0x51f82940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:25.485] < 6275-0x5017f940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 1 ops

[2013-11-28 09:22:25.871] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 3841+100 msgs

[2013-11-28 09:22:25.872] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.872] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.872] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.872] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.872] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.873] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.873] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.873] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.873] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.873] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.874] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.874] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.874] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.874] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.874] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 252 messages (45108 bytes)

[2013-11-28 09:22:25.874] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-PRODUCE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: produce messageset with 161 messages (28819 bytes)

[2013-11-28 09:22:25.874] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 0+0 msgs

[2013-11-28 09:22:25.875] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 18+0 msgs

[2013-11-28 09:22:25.916] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 160+20 msgs

[2013-11-28 09:22:25.957] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-TOPPAR: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: kafka-perf [0] 140+180 msgs

[2013-11-28 09:22:26.065] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BROKERFAIL: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: failed: err: Local: Broker transport failure: (errno: Resource temporarily unavailable)

[2013-11-28 09:22:26.066] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Broker changed state UP -> DOWN

[2013-11-28 09:22:26.066] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-3-FAIL: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: nelo76.nhnchengdu.com:9092/bootstrap: Receive failed: Disconnected

[2013-11-28 09:22:26.066] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BROKERFAIL: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: failed: err: Local: Broker transport failure: (errno: Connection reset by peer)

[2013-11-28 09:22:26.066] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Purging bufq

[2013-11-28 09:22:26.066] < 6275-0x4a776940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/bootstrap: Purging bufq

[2013-11-28 09:22:26.066] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-STATE: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Broker changed state UP -> DOWN

[2013-11-28 09:22:26.066] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-3-FAIL: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: nelo76.nhnchengdu.com:9092/0: Receive failed: Receive error: Connection reset by peer

[2013-11-28 09:22:26.066] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-BUFQ: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: Purging bufq

[2013-11-28 09:22:26.066] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) not delivered

[2013-11-28 09:22:26.066] < 6275-0x5b591940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-MSGSET: rdkafka#producer-0: nelo76.nhnchengdu.com:9092/0: MessageSet with 252 message(s) encountered error: Local: Broker transport failure

[2013-11-28 09:22:26.070] < 6275-0x4c579940> [INFO][kafkaSink.cpp:335] RDKAFKA-7-QSERVE: rdkafka#producer-0: Serving 2 ops

[2013-11-28 09:22:26.070] < 6275-0x4c579940> [INFO][kafkaSink.cpp:558] The broker is dead
[2013-11-28 09:22:26.070] < 6275-0x4c579940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/bootstrap: Receive failed: Disconnected
[2013-11-28 09:22:26.070] < 6275-0x4c579940> [ERROR][kafkaSink.cpp:329] the kafka(rdkafka#producer-0) has error: nelo76.nhnchengdu.com:9092/0: Receive failed: Receive error: Connection reset by peer
[2013-11-28 09:22:56.289] < 6275-0x4bb78940> [INFO][neloFailureTolerant.cpp:246] [Tolerant] Open tolerant file(./cache/thrift-kafka.fail) success.
[2013-11-28 09:22:56.592] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:148] [Tolerant]=====================================Begin==================================
[2013-11-28 09:22:56.592] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:149] [Tolerant] total=78280, left=78280, send=0
[2013-11-28 09:22:56.592] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:150] [Tolerant] file: ./cache/thrift-kafka.fail
[2013-11-28 09:22:56.592] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:151] [Tolerant]============================================================================
[2013-11-28 09:22:56.592] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:150] [Tolerant] file: ./cache/thrift-kafka.fail
[2013-11-28 09:22:56.592] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:151] [Tolerant]============================================================================
[2013-11-28 09:24:57.003] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:179] [Tolerant] total=39039163, left=39035067, send=0
[2013-11-28 09:26:58.006] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:179] [Tolerant] total=77611942, left=77607846, send=0
[2013-11-28 09:27:10.205] < 6275-0x4b177940> [INFO][kafkaSink.cpp:269] ======================Kafka Sink===========================
[2013-11-28 09:27:10.205] < 6275-0x4b177940> [INFO][kafkaSink.cpp:270] Kafka sink deal message count: (529) per second, message size: (25398) Bytes per second, total count:(158739), total size:(7619472)
[2013-11-28 09:27:10.205] < 6275-0x4b177940> [INFO][kafkaSink.cpp:279] Timer job set the broker alive for fail-tolerant thread
[2013-11-28 09:27:10.206] < 6275-0x60599940> [INFO][kafkaSink.cpp:201] Pass one log from fail-tolerant thread to check whether the broker alive
[2013-11-28 09:27:15.613] < 6272-0x42569940> [INFO][processMasterSignal.cpp:145] Monitor thread of master process(6272) is working.
[2013-11-28 09:27:20.623] < 6275-0x42f6a940> [INFO][processChild.cpp:197] Monitor thread of child process(6275) is working.




[2013-11-28 09:28:59.002] < 6275-0x60599940> [INFO][neloFailureTolerant.cpp:179] [Tolerant] total=116649972, left=116645876, send=0

and when I start the kafka again, the connection is established.

tcp        0      0 :::9092                     :::*                        LISTEN      
tcp        0      0 ::ffff:10.34.130.76:59647   ::ffff:10.34.130.76:9092    ESTABLISHED 
tcp        0      0 ::ffff:10.34.130.76:59644   ::ffff:10.34.130.76:9092    TIME_WAIT   
tcp        0      0 ::ffff:10.34.130.76:23902   ::ffff:10.34.130.76:9092    TIME_WAIT   
tcp        0      0 ::ffff:10.34.130.76:9092    ::ffff:10.34.130.76:59647   ESTABLISHED 
tcp        0      0 ::ffff:10.34.130.76:9092    ::ffff:10.34.130.147:39468  ESTABLISHED 
tcp        0      0 ::ffff:10.34.130.76:9092    ::ffff:10.34.130.147:39467  ESTABLISHED

edenhill · 2013-11-29T13:19:13Z

I dont see any problems in the last log output you added here.
It produces messages until the broker is taken down, and the messageset currently in flight fails when the broker is down, which is okay if your message.send.max.retries value is set to 0 (default is 2, in which case it should retry).

What happens after the brokers come up again?
Would be very good with a complete log of what happens before broker takedown, during broker down, and after broker comes up again.
This last log is missing the last part.

Thanks

xiaojun789 · 2013-12-03T08:11:58Z

HI,
I past all the logs here, and when the brokers come up again, there is no other information output from librdkafka, it seems dead.

edenhill · 2013-12-03T08:41:58Z

Okay, thanks for clarifying.

Would it be possible for you to attach gdb to this process when librdkafka seems to be dead:

gdb <yourprogram> <pid>

and then issue the following command in gdb:

thread apply all bt

And paste the output in this issue.

xiaojun789 · 2013-12-03T08:44:22Z

Hi,
I just test it again, when I start/stop the broker about 4 times, the librdkafka has no log information output. it seems work thread is dead. but the logs is to large. please give me your email address, I will send you the logs by email.
Thanks~!

edenhill · 2013-12-03T08:45:18Z

rk_m@edenhill.se

edenhill · 2013-12-03T08:45:44Z

Oh, and I cant seem to see the pictures that you upload to this issue. The links are broken.

xiaojun789 · 2013-12-03T09:02:05Z

I already send you an email.
forget the picture, it already delete that link.

xiaojun789 · 2013-12-03T10:45:40Z

(gdb) thread apply all bt

Thread 52 (Thread 0x416f0940 (LWP 23881)):
#0  0x00000031c80ca366 in poll () from /lib64/libc.so.6
#1  0x0000000000631651 in do_io (v=<value optimized out>) at src/mt_adaptor.c:387
#2  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#3  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x420f1940 (LWP 23882)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000063143b in do_completion (v=<value optimized out>) at src/mt_adaptor.c:463
#2  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#3  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x42af2940 (LWP 23883)):
#0  0x00000031c8c0b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ade6f937c18 in boost::condition_variable::timed_wait(boost::unique_lock<boost::mutex>&, boost::posix_time::ptime const&) ()
   from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.

Thread 49 (Thread 0x434f3940 (LWP 23884)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x7fff3b3de468, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d097f in nana::SignalThread::_threadWorker (this=0x7fff3b3de428) at src/common/utils/utilThread.cpp:367
#3  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#4  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#5  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x43ef4940 (LWP 23885)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176f9eb0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176f9eb0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176f9eb0) at src/common/utils/utilThread.cpp:97
---Type <return> to continue, or q <return> to quit---
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x448f5940 (LWP 23886)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fa780) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176fa780) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fa780) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x452f6940 (LWP 23887)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fac60) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176fac60) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fac60) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x45cf7940 (LWP 23888)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fb330) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176fb330) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fb330) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x466f8940 (LWP 23889)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fba20) at src/common/utils/utilThread.h:249
---Type <return> to continue, or q <return> to quit---
#3  nana::SafeThreadPool::Customer::running (this=0x176fba20) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fba20) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x470f9940 (LWP 23890)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fc110) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176fc110) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fc110) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x47afa940 (LWP 23891)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fc800) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176fc800) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fc800) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x484fb940 (LWP 23892)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fcef0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176fcef0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fcef0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x48efc940 (LWP 23893)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
---Type <return> to continue, or q <return> to quit---
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fd5e0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176fd5e0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fd5e0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

xiaojun789 · 2013-12-03T10:46:27Z

Thread 39 (Thread 0x498fd940 (LWP 23894)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0xb2c510, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x176fdcd0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x176fdcd0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176fdcd0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x4a2fe940 (LWP 23895)):
#0  0x00000031c8c0d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000031c8c08e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000031c8c08d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000066ca8a in rd_kafka_topic_scan_all (rk=0x176fe620, now=16245087237069) at rdkafka_topic.c:843
#4  0x00000000006633a7 in rd_kafka_thread_main (arg=0x176fe620) at rdkafka.c:748
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x4acff940 (LWP 23896)):
#0  0x00000031c8c0d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000031c8c08e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000031c8c08d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000667a8c in rd_kafka_topic_leader_query0 (rk=0x176fe620, rkt=0x0, do_rk_lock=1) at rdkafka_broker.c:1000
#4  0x00000000006691ed in rd_kafka_broker_fail (rkb=0x176febd0, err=RD_KAFKA_RESP_ERR__TRANSPORT, fmt=0x89f8a9 "Receive failed: %s") at rdkafka_broker.c:368
#5  0x0000000000669644 in rd_kafka_recv (rkb=0x176febd0) at rdkafka_broker.c:1266
#6  0x0000000000669e18 in rd_kafka_broker_io_serve (rkb=0x176febd0) at rdkafka_broker.c:2026
#7  0x000000000066b595 in rd_kafka_broker_ua_idle (arg=0x176febd0) at rdkafka_broker.c:2045
#8  rd_kafka_broker_thread_main (arg=0x176febd0) at rdkafka_broker.c:3018
#9  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#10 0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x4b700940 (LWP 23897)):
---Type <return> to continue, or q <return> to quit---
#0  0x00000031c8c0b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ade6f937c18 in boost::condition_variable::timed_wait(boost::unique_lock<boost::mutex>&, boost::posix_time::ptime const&) ()
   from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#2  0x00002ade6f933571 in boost::this_thread::sleep(boost::posix_time::ptime const&) () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#3  0x00000000004dfdfa in sleep (uiMilliSec=<value optimized out>) at /home/jerry/nelo2/include/boost/thread/detail/thread.hpp:347
#4  nana::utilSleep (uiMilliSec=<value optimized out>) at src/common/utils/utilTools.cpp:236
#5  0x00000000005256c8 in nana::TimerThread<NeloSink>::running (this=0x176f6f30) at ./src/common/common/../utils/utilThread.h:606
#6  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176f6f30) at src/common/utils/utilThread.cpp:97
#7  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#8  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#9  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x4c101940 (LWP 23898)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x17701b00) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x17701b00) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x17701b00) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x4cb02940 (LWP 23899)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x17700040) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x17700040) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x17700040) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x4d503940 (LWP 23900)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x177006f0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x177006f0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x177006f0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
---Type <return> to continue, or q <return> to quit---
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x4df04940 (LWP 23901)):
#0  0x00000031c8c0b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ade6f937c18 in boost::condition_variable::timed_wait(boost::unique_lock<boost::mutex>&, boost::posix_time::ptime const&) ()
   from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#2  0x00002ade6f933571 in boost::this_thread::sleep(boost::posix_time::ptime const&) () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#3  0x00000000004dfdfa in sleep (uiMilliSec=<value optimized out>) at /home/jerry/nelo2/include/boost/thread/detail/thread.hpp:347
#4  nana::utilSleep (uiMilliSec=<value optimized out>) at src/common/utils/utilTools.cpp:236
#5  0x000000000051b6ef in NeloFailStoreLocal::running (this=0x176f7a40) at src/common/core/neloFailureTolerant.cpp:185
#6  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176f7a48) at src/common/utils/utilThread.cpp:97
#7  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#8  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#9  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x4e905940 (LWP 23902)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x17706790) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x17706790) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x17706790) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x4f306940 (LWP 23903)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x17706de0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x17706de0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x17706de0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x4fd07940 (LWP 23904)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
---Type <return> to continue, or q <return> to quit---
#2  0x00000000004d7eeb in dequeue (this=0x17707430) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x17707430) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x17707430) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x50708940 (LWP 23905)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x17707a80) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x17707a80) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x17707a80) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x51109940 (LWP 23906)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x177080d0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x177080d0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x177080d0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x51b0a940 (LWP 23907)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x177087a0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x177087a0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x177087a0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x5250b940 (LWP 23908)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x17708e90) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x17708e90) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x17708e90) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x52f0c940 (LWP 23909)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x17709650) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x17709650) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x17709650) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x5390d940 (LWP 23910)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x2aaaac000c60) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x2aaaac000c60) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x2aaaac000c60) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x5430e940 (LWP 23911)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x2aaaac001760) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x2aaaac001760) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x2aaaac001760) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 21 (Thread 0x54d0f940 (LWP 23912)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x1770d340) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x1770d340) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x1770d340) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x55710940 (LWP 23913)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x1770dcf0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x1770dcf0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x1770dcf0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x56111940 (LWP 23914)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x1770e340) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x1770e340) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x1770e340) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x56b12940 (LWP 23915)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x1770b800) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x1770b800) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x1770b800) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x57513940 (LWP 23916)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x1770bd40) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x1770bd40) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x1770bd40) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x57f14940 (LWP 23917)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x1770c4a0) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x1770c4a0) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x1770c4a0) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x58915940 (LWP 23918)):
#0  0x00000031c8c0af59 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004d6cd9 in boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > (this=0x176fff80, m=...)
    at /home/jerry/nelo2/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00000000004d7eeb in dequeue (this=0x1770cb70) at src/common/utils/utilThread.h:249
#3  nana::SafeThreadPool::Customer::running (this=0x1770cb70) at src/common/utils/utilThread.h:421
#4  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x1770cb70) at src/common/utils/utilThread.cpp:97
#5  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#6  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#7  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x59316940 (LWP 23919)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17712990, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17712990, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x17710d20) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x59d17940 (LWP 23920)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17713260, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17713260, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x17710fc0) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x5a718940 (LWP 23921)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17713b30, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17713b30, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x17711260) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x5b119940 (LWP 23922)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17714400, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17714400, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x17711550) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

xiaojun789 · 2013-12-03T10:46:50Z

Thread 10 (Thread 0x5bb1a940 (LWP 23923)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17714db0, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17714db0, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x177117f0) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x5c51b940 (LWP 23924)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17715790, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17715790, flags=0) at event.c:1575
---Type <return> to continue, or q <return> to quit---
#3  0x000000000055473f in IOService::_workerThread (this=0x17711b20) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x5cf1c940 (LWP 23925)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17716160, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17716160, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x17711dc0) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x5d91d940 (LWP 23926)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17716b40, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17716b40, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x17712060) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x5e31e940 (LWP 23927)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17717510, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17717510, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x17712320) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x5ed1f940 (LWP 23928)):
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17717ef0, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17717ef0, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x177126f0) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x5f720940 (LWP 23929)):
---Type <return> to continue, or q <return> to quit---
#0  0x00000031c80d3648 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000618ca4 in epoll_dispatch (base=0x17718ac0, tv=<value optimized out>) at epoll.c:407
#2  0x000000000060a9f9 in event_base_loop (base=0x17718ac0, flags=0) at event.c:1575
#3  0x000000000055473f in IOService::_workerThread (this=0x177109d0) at src/common/service/eventPlusWorker.cpp:39
#4  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#5  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#6  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x60121940 (LWP 23930)):
#0  0x00000031c8c0b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ade6f937c18 in boost::condition_variable::timed_wait(boost::unique_lock<boost::mutex>&, boost::posix_time::ptime const&) ()
   from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#2  0x00002ade6f933571 in boost::this_thread::sleep(boost::posix_time::ptime const&) () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#3  0x00000000004dfdfa in sleep (uiMilliSec=<value optimized out>) at /home/jerry/nelo2/include/boost/thread/detail/thread.hpp:347
#4  nana::utilSleep (uiMilliSec=<value optimized out>) at src/common/utils/utilTools.cpp:236
#5  0x0000000000535429 in nana::TimerThread<NeloSource>::running (this=0x176f68b0) at ./src/common/common/../utils/utilThread.h:606
#6  0x00000000004cf078 in nana::SafeThread::_threadWorker (this=0x176f68b0) at src/common/utils/utilThread.cpp:97
#7  0x00002ade6f932b8f in thread_proxy () from /home/jerry/nelo2/collector/bin/../../lib/libboost_thread.so.1.49.0
#8  0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#9  0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x60b22940 (LWP 23931)):
#0  0x00000031c8c0d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000031c8c08e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000031c8c08d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000667a17 in rd_kafka_broker_any (rk=0x176fe620, rkt=0x176fe3f0, do_rk_lock=1) at rdkafka_broker.c:978
#4  rd_kafka_topic_leader_query0 (rk=0x176fe620, rkt=0x176fe3f0, do_rk_lock=1) at rdkafka_broker.c:1001
#5  0x0000000000669b89 in rd_kafka_produce_msgset_reply (rkb=0x2aaaac003ec0, err=RD_KAFKA_RESP_ERR__TRANSPORT, reply=0x0, request=0x2aaab00008c0, opaque=0x2aaaac0047b0)
    at rdkafka_broker.c:1523
#6  0x00000000006651ca in rd_kafka_bufq_purge (rkb=0x2aaaac003ec0, rkbufq=<value optimized out>, err=RD_KAFKA_RESP_ERR__TRANSPORT) at rdkafka_broker.c:244
#7  0x00000000006690f9 in rd_kafka_broker_fail (rkb=0x2aaaac003ec0, err=RD_KAFKA_RESP_ERR__TRANSPORT, fmt=0x89f8a9 "Receive failed: %s") at rdkafka_broker.c:341
#8  0x0000000000669644 in rd_kafka_recv (rkb=0x2aaaac003ec0) at rdkafka_broker.c:1266
#9  0x0000000000669e18 in rd_kafka_broker_io_serve (rkb=0x2aaaac003ec0) at rdkafka_broker.c:2026
#10 0x000000000066a668 in rd_kafka_broker_producer_serve (rkb=0x2aaaac003ec0) at rdkafka_broker.c:2146
#11 0x000000000066b648 in rd_kafka_broker_thread_main (arg=0x2aaaac003ec0) at rdkafka_broker.c:3020
#12 0x00000031c8c0677d in start_thread () from /lib64/libpthread.so.0
#13 0x00000031c80d325d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2ade7261d9f0 (LWP 23879)):
#0  0x00000031c8031638 in do_sigtimedwait () from /lib64/libc.so.6
#1  0x00000031c80316fe in sigtimedwait () from /lib64/libc.so.6
#2  0x000000000047b9da in ChildProcess::_childSignalCycle (this=0x7fff3b3de230, uiMillSecond=<value optimized out>) at src/common/process/processChild.cpp:653
---Type <return> to continue, or q <return> to quit---
#3  0x000000000047dad7 in ChildProcess::childProcessCycle (this=0x7fff3b3de230) at src/common/process/processChild.cpp:714
#4  0x000000000048b527 in MasterProcess::_forkChildProcess (this=0xb294e0) at src/common/process/processMaster.cpp:393
#5  0x00000000004919ed in MasterProcess::runningService (argc=3, argv=0x7fff3b3deb88, eModel=<value optimized out>) at src/common/process/processMaster.cpp:546
#6  0x00000031c801d994 in __libc_start_main () from /lib64/libc.so.6
#7  0x00000000004223f9 in _start ()

edenhill · 2013-12-03T13:32:48Z

Jerry, can you update to the latest version on master branch and see if you can reproduce?
This issue should be fixed now by commit 2c2370c

xiaojun789 · 2013-12-04T02:00:42Z

HI,
I test your latest code, and restart the kafka about 10 times, It seems work ok. I should do more test.
Thanks~!

edenhill · 2013-12-04T07:18:30Z

Thanks for your help in sorting out this issue!

ghost assigned edenhill Nov 21, 2013

edenhill added a commit that referenced this issue Dec 3, 2013

Fixed recursive locking on broker fail+buf retry (issue #27)

2c2370c

edenhill closed this as completed Dec 4, 2013

DEvil0000 mentioned this issue Oct 12, 2015

rdkafka stopping connection retries on RdKafka::ERR__ALL_BROKERS_DOWN #373

Closed

MockingJayWong mentioned this issue Jul 9, 2019

rd_kafka_topic_new after using admin api #2399

Closed

7 tasks

Furuta-Masakazu-quick mentioned this issue Sep 29, 2020

Producer detects "All broker connections are down" when rolling restarts #3090

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Does the library support broker recover back #27

Does the library support broker recover back #27

xiaojun789 commented Nov 21, 2013

edenhill commented Nov 21, 2013

xiaojun789 commented Nov 27, 2013

edenhill commented Nov 27, 2013

xiaojun789 commented Nov 27, 2013

edenhill commented Nov 27, 2013

xiaojun789 commented Nov 28, 2013

xiaojun789 commented Nov 28, 2013

edenhill commented Nov 29, 2013

xiaojun789 commented Dec 3, 2013

edenhill commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

edenhill commented Dec 3, 2013

edenhill commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

edenhill commented Dec 3, 2013

xiaojun789 commented Dec 4, 2013

edenhill commented Dec 4, 2013

Does the library support broker recover back #27

Does the library support broker recover back #27

Comments

xiaojun789 commented Nov 21, 2013

edenhill commented Nov 21, 2013

xiaojun789 commented Nov 27, 2013

edenhill commented Nov 27, 2013

xiaojun789 commented Nov 27, 2013

edenhill commented Nov 27, 2013

xiaojun789 commented Nov 28, 2013

xiaojun789 commented Nov 28, 2013

edenhill commented Nov 29, 2013

xiaojun789 commented Dec 3, 2013

edenhill commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

edenhill commented Dec 3, 2013

edenhill commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

xiaojun789 commented Dec 3, 2013

edenhill commented Dec 3, 2013

xiaojun789 commented Dec 4, 2013

edenhill commented Dec 4, 2013