[STORM-2713] when the connection to the first zkserver is timeout,storm-kafka's kafkaspout will throw a exception

[liu-zhaokun]

https://issues.apache.org/jira/browse/STORM-2713
when the connection to the first zkserver is timeout,storm-kafka's kafkaspout will throw a exception without attempting to connect other zkserver,even zk can also work with one node down.

[HeartSaVioR]

Which value do you think this method returns when exception is caught, which action caller should take? Without proper action for failing back, it is better to throw exception and fail-fast.

[HeartSaVioR]

And compilation fails on the change. Please see the result of Travis build.

[liu-zhaokun]

@HeartSaVioR
When exception is caught,we should only print the error messages,so that users can be aware of there are some problems in their cluster such as one role of zk was down.

[liu-zhaokun]

@HeartSaVioR
For example,one role of zk is down in users cluster,storm-kafka will throw a exception,even storm also can connect another zk role and works.I will check the result of Travis build.

[HeartSaVioR]

When RuntimeException is propagated to call hierarchy, some place will leave a error message. (it is guaranteed when worker is crashing)

Btw, I guess you may need to think and try to answer my questions above by yourself.
Please note that core principal of Storm is 'fail-fast', because in many case things are not restorable and retryable, and we had to kill the process to prevent process to go with unexpected behavior.

[HeartSaVioR]

That may be viable and possible, but you may need to add retry logic rather than just removing throw statement.

[liu-zhaokun]

I think it doesn't affect 'fail-fast',I just want to guarantee users' Topology can work normally.

[liu-zhaokun]

@HeartSaVioR
I mean topology should work even we can't connect one of the zkservers,because the data is shared among zkservers. But if we throw a exception,storm will not attempt to connect other zkserver,and exit.

[srdo]

I think Curator is supposed to handle retry and partial connection loss to the Zookeeper cluster for us. I'd rather we use Curator for this instead of trying to add our own retry loops around it.

[liu-zhaokun]

@srdo
I don't think it's related to retry, the key is how storm will do when the first zkserver is down.

[srdo]

@liu-zhaokun Yes, that's what I'm talking about. If the first zkserver is down, I think Curator is supposed to try another server in the connection string.

[liu-zhaokun]

@srdo
I guess you support my PR,right?

[srdo]

@liu-zhaokun No, not as it is currently. I think Curator already handles failing over to another Zookeeper server for us, so if the user passes a zkStr with multiple servers here https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java#L61 and sets the retry policy at https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java#L64 to a few attempts, then I don't see why we need to add another layer of retries on top of that?

Plus as @HeartSaVioR points out, this doesn't compile.

[liu-zhaokun]

@srdo
Thanks for your reply.I mean if curator has retried several times and failed,the code where I modify will throw a exception,and it will never attempt to connect another zkserver. Could you help me solve doubts?
I will check the CI later.