FailedToRebalanceConsumerError: Exception: NODE_EXISTS[-110] #449
Comments
|
Looks like it failed to register the consumer with zookeeper. Run your process with env |
|
I had the same issue. The doc says it is the problem of FailedToRebalanceConsumerError: Exception: NODE_EXISTS[-110] @m2studio How did u start your program with forever? In my case, I started 20 instances of node threads with a shell script and random error occur within those 20 instances. But if I started those instances with a few time delayed for each threads would solve this problem. I read about debug logs with kafka-node:* turned on. Seems kafka would rebalance partitions when ever a new thread joins into the party. So that if we started threads within a short amount of time. Rebalance could lead to an error. The following are some debug logs: Hope this could help you. |
|
A stack trace of the Also have you tried using the new Consumer Group? It's very similar to HLC and should not suffer from these |
|
@hyperlink My kafka version is 0.8.2 so that I can't use Consumer Group. Gonna have a deep look at this issue later. |
|
Hi @m2studio @gogorush, have you found any solution regarding this issue? We are currently facing the same problem in a prod environment, and we are struggling to find a way to solve it. Any hint towards where to look into would be appreciated |
|
This is caused by the rebalance issue when more consumers added into the network. I upgraded my kafka to 0.9+ and use the new ConsumerGroup API to avoid this issue. I checked this issue earlier, conflunt(the company behind kafka) posted an article about their new consumer in 0.9 said and I quote 'This new consumer also adds a set of protocols for managing fault-tolerant groups of consumer processes. Previously this functionality was implemented with a thick Java client (that interacted heavily with Zookeeper). The complexity of this logic made it hard to build fully featured consumers in other languages'. SO I guess Javascript is one of the languages. And for many other reasons, I strongly recommend you to upgrade your kafka to 0.9+ too. Check our more on this link Hope this could help you @s-rodriguez . |
|
Wow, thanks a lot @gogorush! I'll definitely check out kafka 0.9+ with the new ConsumerGroup |
I have a process to consumer messages from Kafka 24/7 and use forever on top to ensure my process will never stop and 100% alive.
But once I let my process run a few hours, I found that it stopped to consumer message (my process is still running but somehow there is no message consuming). Then I found these errors in my log file
FailedToRebalanceConsumerError: Exception: NODE_EXISTS[-110]
FailedToRegisterConsumerError: Consumer not registered prerender-live-group_186636ca-3ccb-4660-9931-6f4160e71fdb
FailedToRegisterConsumerError: Path wasn't created
my code (if you want more detail to investigate please let me know)
this.client = new kafka.Client(this.zookeeperServer); this.consumer = new kafka.HighLevelConsumer(this.client, [{ topic: this.topicName }], this.options); this.consumer.on('message', this.onMessage.bind(this)); this.consumer.on('error', this.onError.bind(this));I have tried to find the root cause since last 2 weeks. But I'm still not able to solve it, even I get the latest version (0.5.5).
Please help me, I need the process to survive all day all night.
my configuration
"options": { "autoCommit": true, "autoCommitMsgCount": 1, "autoCommitIntervalMs": 300, "fromOffset": false, "groupId": "new-oad-group", "fromBeginning": false, "fetchMaxWaitMs": 100, "fetchMaxBytes": 1048576, "maxTickMessages": 100, "encoding": "utf8" }Machine info
OS ==> 2.6.32-573.26.1.el6.centos.plus.x86_64
CPU(s) 8
MemTotal: 15300380 kB
Last test
-- first topic has 80 partitions
-- second topic has 6 partitions
The text was updated successfully, but these errors were encountered: