output plugins load balancing via hosts array not aware of other logstash instances.

[kobybr]

• logstash-2.4.0
• centos 7.2

Say I have three logstash servers loadbalanced behind an F5. All three servers are identical, and contain an elasticsearch output plugin with a host array of something like: hosts => ["host1","host2",host3","host4"].

What I'm trying to prevent is a scenerio where the three logstash servers would get in a rhythm of rotating through the same hosts at the same time, thus swamping one elasticsearch host, while the others go unused.

From what I understand, an individual instance will round-robin through the array of available elasticsearch hosts. I believe I have experienced this scenario and was likely the cause of a backlog of unprocessed/dropped udp packets. I alleviated the issue by having each logstash instance only contain unique sets of hosts. i.e. logstash-1 would contain hosts => ["host1","host2"] and logstash-2 would contain hosts => ["host3","host4"]. This would not be the most ideal way of handling.

Am I missing something? Thoughts?

Thanks

[jsvd]

a connection is picked per bulk request so it's unlikely that three instances an any point will be hammering the same server for a large period of time unless each bulk request has very large documents (in the order of megabytes) or the ids for events are being set explicitly are aren't well balanced causing hotspots due to routing issues.

I'm closing this issue as I don't believe this is an issue, please reopen or open a new one if you're seeing a similar problem