Thundering herd when starting many Beats at the same time.

[jarpy]

The timing of Beats is so accurate that they create bursty traffic when simultaneously restarted. In a large fleet with good configuration management, it's feasible that many hundreds of Beats could be restarted within one second of each other. They then proceed to stay in perfect sync.

Perhaps an intial startup delay of rand(period) would be nice here?

[andrewkroh]

Since the earliest implementing I was thinking that the scheduling of each individual metricset should be staggered at startup to help smooth the CPU load on a host. I hadn't consider the herd effect caused by an entire fleet restarting. The same problem will affect Beats when central monitoring is available and you can reconfigure all at once.

I think both issues will be addressed if we introduce a random delay into the startup of each metricset. Thanks for providing a visualization of the issue. We can check this again after introducing the a fix for this.

[tsg]

+1 for random delay at startup.