Protecting cluster in elastic search from big queries

[nitin2goyal]

PROBLEM

We have use case where we are running ES on 100s of nodes and we get a mix of long and small running concurrent queries, where long and short is defined by number of shards it executes on.

https://discuss.elastic.co/t/long-query-starving-other-small-queries-and-slowing-down-the-cluster/163665?u=dextermorgan

I know we have a feature where we can limit the number of concurrent shards per search request and protect our cluster form one big query but what if there are more concurrent big queries with small queries. It would still starve the small queries

PROPOSAL

I propose to introduce a new search queue, maybe called as search_fast with configurable threads and elasticsearch can schedule smaller queries on this queue in case "search" thread queue size is large.

Let me know your thoughts regarding this and I can come up with a design and later, pull request.

P.S. - Something similar in YARN - https://hortonworks.com/blog/yarn-capacity-scheduler/

[elasticmachine]

Pinging @elastic/es-search

[jpountz]

Hi @nitin2goyal, thanks for reporting this feature request. This has been a recurring request (with some variations) so we decided to start a new thread that would collect feedback regarding assigning priorities to search requests: #37867. Please feel free to give more details about your use-case there.