Add priority to the threadpool

[gmoskovicz]

When a bulk request is sent to elasticsearch, if there are other bulk operations simultaneously being sent there is no way to control how they will be handle, hence the requests will be handled randomly/automatically.

My proposal/question is about adding a priority parameter, which will be useful to set at the top of the queue/priority list those bulk (or any other requests) that are more important that the once that are being handled at some point.

[jpountz]

Using a priority queue could help have some entries be taken out of the queue before others, but in the case that the problem is that the threadpool is already full with long-running bulk requests, this would not help as we would still have to wait for these long-running bulk requests to finish before being able to take a new one.

An alternative would be to have several threadpools but I don't like it as elasticsearch already creates hundreds of threads by default (see eg. #12666).

[clintongormley]

If your bulk queue is full, then having a priority just indicates which indexing requests you want to throw away first/last. This doesn't make much sense to me.

[ningjun]

I like the idea of having several bulk thread pools. This way user can set a dedicated thread pool to handle high priority bulk request.
This is a very useful feature and I think many customers would like to have this feature.

[Habitats]

Any progress on this?

[emilyfcy]

Is it possible to add priority to the search thread pool? We ran into the case that some of the search take long time, for example deep paging. Searching that in parallel might block all the rest search requests. So, it would be nice if we have limited low priority search pool to constraint the output of some searches that allows to be slow.

[jpountz]

It is not possible. Plus I'd like to point out that fixing this is the search thread pool is not enough. For instance if an ongoing request is stealing all the I/O capacity of the machine, then other requests will be slow, regardless of what is done at the threadpool level.

[emilyfcy]

OK. If the bottleneck comes to CPU, it would be helpful. If it comes to others, like IO, cache, setting low priority on the threads might not be useful.

[clintongormley]

@emilyfcy See the max_concurrent_shard_requests setting which limits the number of concurrent shards that are searched by a single search request. This could fit your need.

[dnhatn]

This feature request is an interesting idea but since its opening we have not seen enough feedback that it is a feature we should pursue. We prefer to close this issue as a clear indication that we are not going to work on this at this time. We are always open to reconsidering this in the future based on compelling feedback; despite this issue being closed please feel free to leave feedback on the proposal (including +1s).

[elasticmachine]

Pinging @elastic/es-core-infra