[Alerting] Sometimes, when there are multiple Kibana, some instances fail to pick up any tasks

[gmmorris]

We've identified an issue where multiple Kibana running in parallel can clash in a manner that forces some of them to skip all work.
We're not sure how to recreate this consistently... but it seems to be a clash during the UpdateByQuery step,

We think this explains the drift that we see at times climb to 20s+

Easiest way to recreate this is by running our perf tests at a high rate (1200+ alerts per minute) with at least 6 Kibana, though it can happen with less Kibana at times, it is less likely.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[gmmorris]

We've confirmed the root cause is most likely version conflicts, so a new PR is being reviewed now which shifts the polling interval by a random amount to redistribute the polling when needed.

[gmmorris]

32 Kibana, all balancing work, clashes are addressed by shifting polling.