SearchRequest cancellation based on timeout

[sohami]

Feature request

In ES-7.4 version, support for cancelling a search request when the client connection is closed is added. Also the current timeout parameter in the search request is not the request level timeout, instead it is checked at shard level search request of each batch. A search request on a coordinator can span to multiple batches of ShardSearchTransportRequest when throttleConcurrentRequest is enabled. This means when there are 2 such batches, then timeout will be applied separately on the shard request in these 2 batches and actual request may end up timing out after 2 x timeout parameter value. For an index with large shard count, the number of batches can be higher and the request will timeout after long delay.

I am wondering if we can provide a new search request parameter which can be enforced at the parent request level (e.g. cancel_after_timeinterval), such that after expiry it will cancel the search request. This will be helpful for cases where the connection is still alive but user is running a large resource hungry search and doesn't wait until completion. With the new parameter which can be set at request level, user can set it as per their workload/use-case and if query doesn't complete by then, it will be cancelled in the backend. This will help to save the unnecessary resource the query was consuming for which user is not interested in the results anymore.
Also there can be a cluster level parameter for this as well such that if a user is not willing to set it at per request then the cluster level parameter can be used to cancel all the search request (if set) after the timeout. Order of precedence will be: requestParameter > clusterParameter. If none of the parameters are set then behavior will fallback to what it is today.

Please let me know your thoughts about this new feature.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[javanna]

There have been multiple discussions around the search timeout, its meaning, current behaviour and improvements that could be made: see #47716 , #54056 and #30897 . I think this issue is proposing something very similar if not the same as #30897 specifically.

Also, I would like to add that the async search API was recently added, which I think would help in the scenario you described: you can submit an async search, wait e.g. 10 seconds to see if you get results, then start polling through get async search for new results and also cancel the search at your own convenience using delete async search.

[sohami]

@javanna - Thanks for your response. Yes my ask is same as #30897. But looks like that implementation is not part of ES and is rather done by @fantapsody in their private branch. Is there any plan to use that recommendation and implement in ES itself ?

[javanna]

ok I hope you don't mind then if I am closing this in favour of #30897.