Option to add safeguard to prevent overly broad time filters that hit too many indices

[ppf2]

Use case is that there can be a number of clusters with many indices (>100 easily) fronted by the tribe node (or simply many indices in a single cluster). The number of indices queried today from Kibana is driven by the time range filter the user selects. There can be situations when the end user selects an overly broad time range (mistakenly, or simply not knowing the consequences of doing so), and cause the query to hit hundreds of indices (or maybe thousands of shards), resulting in poor performing queries that can add stress to the ES clusters.

One idea here is to add a safeguard option to limit the # of indices queried and block users from doing so. We are already doing some prep work using the field stats api to come up with a constrained set of indices to query based on the time filter selected by the end user. This can potentially be a safeguard we can build on top of that field stats check, eg. allow admins to configure a hard limit for the maximum number of indices a query can hit, and once we get the list of indices back from the field stats api, we can strip off the “extra” indices that exceed the threshold. In terms of which indices to return when the threshold is exceeded, maybe the “latest” N number of indices? Or give the user a few options when setting the threshold. Certainly, this will apply mostly for the time based indexing use case.