Async search #49091

javanna · 2019-11-14T14:00:27Z

This is a meta issue to monitor progress around adding support for asynchronous search.

The main goal is to launch a search request and, instead of waiting for its final response, let it run asynchronously while being able to monitor its progress. The progress of an async search includes how many operations have been performed compared to the total number of operations required for completion, as well as intermediate results that may be available before completion.

The high-level steps required for this feature are listed as follows, together with their corresponding licensing:

add ability to control the order in which shards are queried by the search API (OSS) Pre-sort shards based on the max/min value of the primary sort field #49092
add ability to monitor progress of a search request through a listener that gets notified whenever new shard results are available (OSS) Add a listener to track the progress of a search request locally #49471
introduce new endpoints to submit, get and delete an async search (Basic) Add new x-pack endpoints to track the progress of a search asynchronously #49931
Add tests for the AsyncSearchMaintenanceService
Add documentation for the APIs
Add support for the HLRC.

The resiliency of async search is besides the scope of the initial implementation. If the coordinating node goes down while an async search is running, the most recent intermediate results should be made available for retrieval, but the async search is aborted and never retried, hence it needs to be restarted manually.

elasticmachine · 2019-11-14T14:00:29Z

Pinging @elastic/es-search (:Search/Search)

This change automatically pre-sort search shards on search requests that use a primary sort based on the value of a field. When possible, the can_match phase will extract the min/max (depending on the provided sort order) values of each shard and use it to pre-sort the shards prior to running the subsequent phases. This feature can be useful to ensure that shards that contain recent data are executed first so that intermediate merge have more chance to contain contiguous data (think of date_histogram for instance) but it could also be used in a follow up to early terminate sorted top-hits queries that don't require the total hit count. The latter could significantly speed up the retrieval of the most/least recent documents from time-based indices. I took two shortcuts here: * I reused the can_match phase to add the required information for the shard sort. We could instead introduce a new phase but it make sense to me to use the existing phase to add more informations as long as the additional ops are lightweight. * The shard sort is done automatically if the primary search sort is based on a field. However this sorting only makes sense if the range of values in each shard doesn't overlap (time-based indices sorted on timestamp for instance). We could add a new option to enable/disable this behavior or even add an additional `shard_sort` criteria but I also like the fact that users don't need to set any option to benefit from this feature. Relates elastic#49091

…49092) This change automatically pre-sort search shards on search requests that use a primary sort based on the value of a field. When possible, the can_match phase will extract the min/max (depending on the provided sort order) values of each shard and use it to pre-sort the shards prior to running the subsequent phases. This feature can be useful to ensure that shards that contain recent data are executed first so that intermediate merge have more chance to contain contiguous data (think of date_histogram for instance) but it could also be used in a follow up to early terminate sorted top-hits queries that don't require the total hit count. The latter could significantly speed up the retrieval of the most/least recent documents from time-based indices. Relates #49091

This commit adds a function in NodeClient that allows to track the progress of a search request locally. Progress is tracked through a SearchProgressListener that exposes query and fetch responses as well as partial and final reduces. This new method can be used by modules/plugins inside a node in order to track the progress of a local search request. Relates elastic#49091

…49092) This change automatically pre-sort search shards on search requests that use a primary sort based on the value of a field. When possible, the can_match phase will extract the min/max (depending on the provided sort order) values of each shard and use it to pre-sort the shards prior to running the subsequent phases. This feature can be useful to ensure that shards that contain recent data are executed first so that intermediate merge have more chance to contain contiguous data (think of date_histogram for instance) but it could also be used in a follow up to early terminate sorted top-hits queries that don't require the total hit count. The latter could significantly speed up the retrieval of the most/least recent documents from time-based indices. Relates #49091

) This commit adds a function in NodeClient that allows to track the progress of a search request locally. Progress is tracked through a SearchProgressListener that exposes query and fetch responses as well as partial and final reduces. This new method can be used by modules/plugins inside a node in order to track the progress of a local search request. Relates #49091

…stic#49471) This commit adds a function in NodeClient that allows to track the progress of a search request locally. Progress is tracked through a SearchProgressListener that exposes query and fetch responses as well as partial and final reduces. This new method can be used by modules/plugins inside a node in order to track the progress of a local search request. Relates elastic#49091

) (#49691) This commit adds a function in NodeClient that allows to track the progress of a search request locally. Progress is tracked through a SearchProgressListener that exposes query and fetch responses as well as partial and final reduces. This new method can be used by modules/plugins inside a node in order to track the progress of a local search request. Relates #49091

@timestamp

…usly (#49931) ### High level view This change introduces a new API in x-pack basic that allows to track the progress of a search. Users can submit an asynchronous search through a new endpoint called `_async_search` that works exactly the same as the `_search` endpoint but instead of blocking and returning the final response when available, it returns a response after a provided `wait_for_completion` time. ```` # Submit an _async_search and waits up to 100ms for a final response GET my_index_pattern*/_async_search?wait_for_completion=100ms { "aggs": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1h" } } } ```` If after 100ms the final response is not available, a `partial_response` is included in the body: ```` { "id": "9N3J1m4BgyzUDzqgC15b", "version": 1, "is_running": true, "is_partial": true, "response": { "_shards": { "total": 100, "successful": 5, "failed": 0 }, "total_hits": { "value": 1653433, "relation": "eq" }, "aggs": { ... } } } ```` The partial response contains the total number of requested shards, the number of shards that successfully returned and the number of shards that failed. It also contains the total hits as well as partial aggregations computed from the successful shards. To continue to monitor the progress of the search users can call the get `_async_search` API like the following: ```` GET _async_search/9N3J1m4BgyzUDzqgC15b/?wait_for_completion=100ms ```` That returns a new response that can contain the same partial response than the previous call if the search didn't progress, in such case the returned `version` should be the same. If new partial results are available, the version is incremented and the `partial_response` contains the updated progress. Finally if the response is fully available while or after waiting for completion, the `partial_response` is replaced by a `response` section that contains the usual _search response: ```` { "id": "9N3J1m4BgyzUDzqgC15b", "version": 10, "is_running": false, "response": { "is_partial": false, ... } } ```` ## Persistency Asynchronous search are stored in a restricted index called `.async-search` if they survive (still running) after the initial submit. Each request has a keep alive that defaults to 5 days but this value can be changed/updated any time: ````` GET my_index_pattern*/_async_search?wait_for_completion=100ms&keep_alive=10d ````` The default can be changed when submitting the search, the example above raises the default value for the search to `10d`. ````` GET _async_search/9N3J1m4BgyzUDzqgC15b/?wait_for_completion=100ms&keep_alive=10d ````` The time to live for a specific search can be extended when getting the progress/result. In the example above we extend the keep alive to 10 more days. A background service that runs only on the node that holds the first primary shard of the `async-search` index is responsible for deleting the expired results. It runs every hour but the expiration is also checked by running queries (if they take longer than the keep_alive) and when getting a result. Like a normal `_search`, if the http channel that is used to submit a request is closed before getting a response, the search is automatically cancelled. Note that this behavior is only for the submit API, subsequent GET requests will not cancel if they are closed. ## Resiliency Asynchronous search are not persistent, if the coordinator node crashes or is restarted during the search, the asynchronous search will stop. To know if the search is still running or not the response contains a field called `is_running` that indicates if the task is up or not. It is the responsibility of the user to resume an asynchronous search that didn't reach a final response by re-submitting the query. However final responses and failures are persisted in a system index that allows to retrieve a response even if the task finishes. ```` DELETE _async_search/9N3J1m4BgyzUDzqgC15b ```` The response is also not stored if the initial submit action returns a final response. This allows to not add any overhead to queries that completes within the initial `wait_for_completion`. ## Security The `.async-search` index is a restricted index (should be migrated to a system index in +8.0) that is accessible only through the async search APIs. These APIs also ensure that only the user that submitted the initial query can retrieve or delete the running search. Note that admins/superusers would still be able to cancel the search task through the task manager like any other tasks. Relates #49091 Co-authored-by: Luca Cavanna <javanna@users.noreply.github.com>

This commit adds a new AsyncSearchClient to the High Level Rest Client which initially supporst the submitAsyncSearch in its blocking and non-blocking flavour. Also adding client side request and response objects and parsing code to parse the xContent output of the client side AsyncSearchResponse together with parsing roundtrip tests and a simple roundtrip integration test. Relates to elastic#49091

@timestamp

…usly (#49931) (#53591) This change introduces a new API in x-pack basic that allows to track the progress of a search. Users can submit an asynchronous search through a new endpoint called `_async_search` that works exactly the same as the `_search` endpoint but instead of blocking and returning the final response when available, it returns a response after a provided `wait_for_completion` time. ```` GET my_index_pattern*/_async_search?wait_for_completion=100ms { "aggs": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1h" } } } ```` If after 100ms the final response is not available, a `partial_response` is included in the body: ```` { "id": "9N3J1m4BgyzUDzqgC15b", "version": 1, "is_running": true, "is_partial": true, "response": { "_shards": { "total": 100, "successful": 5, "failed": 0 }, "total_hits": { "value": 1653433, "relation": "eq" }, "aggs": { ... } } } ```` The partial response contains the total number of requested shards, the number of shards that successfully returned and the number of shards that failed. It also contains the total hits as well as partial aggregations computed from the successful shards. To continue to monitor the progress of the search users can call the get `_async_search` API like the following: ```` GET _async_search/9N3J1m4BgyzUDzqgC15b/?wait_for_completion=100ms ```` That returns a new response that can contain the same partial response than the previous call if the search didn't progress, in such case the returned `version` should be the same. If new partial results are available, the version is incremented and the `partial_response` contains the updated progress. Finally if the response is fully available while or after waiting for completion, the `partial_response` is replaced by a `response` section that contains the usual _search response: ```` { "id": "9N3J1m4BgyzUDzqgC15b", "version": 10, "is_running": false, "response": { "is_partial": false, ... } } ```` Asynchronous search are stored in a restricted index called `.async-search` if they survive (still running) after the initial submit. Each request has a keep alive that defaults to 5 days but this value can be changed/updated any time: ````` GET my_index_pattern*/_async_search?wait_for_completion=100ms&keep_alive=10d ````` The default can be changed when submitting the search, the example above raises the default value for the search to `10d`. ````` GET _async_search/9N3J1m4BgyzUDzqgC15b/?wait_for_completion=100ms&keep_alive=10d ````` The time to live for a specific search can be extended when getting the progress/result. In the example above we extend the keep alive to 10 more days. A background service that runs only on the node that holds the first primary shard of the `async-search` index is responsible for deleting the expired results. It runs every hour but the expiration is also checked by running queries (if they take longer than the keep_alive) and when getting a result. Like a normal `_search`, if the http channel that is used to submit a request is closed before getting a response, the search is automatically cancelled. Note that this behavior is only for the submit API, subsequent GET requests will not cancel if they are closed. Asynchronous search are not persistent, if the coordinator node crashes or is restarted during the search, the asynchronous search will stop. To know if the search is still running or not the response contains a field called `is_running` that indicates if the task is up or not. It is the responsibility of the user to resume an asynchronous search that didn't reach a final response by re-submitting the query. However final responses and failures are persisted in a system index that allows to retrieve a response even if the task finishes. ```` DELETE _async_search/9N3J1m4BgyzUDzqgC15b ```` The response is also not stored if the initial submit action returns a final response. This allows to not add any overhead to queries that completes within the initial `wait_for_completion`. The `.async-search` index is a restricted index (should be migrated to a system index in +8.0) that is accessible only through the async search APIs. These APIs also ensure that only the user that submitted the initial query can retrieve or delete the running search. Note that admins/superusers would still be able to cancel the search task through the task manager like any other tasks. Relates #49091 Co-authored-by: Luca Cavanna <javanna@users.noreply.github.com>

Relates to elastic#49091

This commit adds a new AsyncSearchClient to the High Level Rest Client which initially supporst the submitAsyncSearch in its blocking and non-blocking flavour. Also adding client side request and response objects and parsing code to parse the xContent output of the client side AsyncSearchResponse together with parsing roundtrip tests and a simple roundtrip integration test. Relates to #49091

This commit adds the "_async_searhc" get and delete APIs to the AsyncSearchClient in the High Level Rest Client. Relates to elastic#49091

This commit adds a new AsyncSearchClient to the High Level Rest Client which initially supporst the submitAsyncSearch in its blocking and non-blocking flavour. Also adding client side request and response objects and parsing code to parse the xContent output of the client side AsyncSearchResponse together with parsing roundtrip tests and a simple roundtrip integration test. Relates to elastic#49091 Backport of elastic#53592

This commit adds a new AsyncSearchClient to the High Level Rest Client which initially supporst the submitAsyncSearch in its blocking and non-blocking flavour. Also adding client side request and response objects and parsing code to parse the xContent output of the client side AsyncSearchResponse together with parsing roundtrip tests and a simple roundtrip integration test. Relates to #49091 Backport of #53592

Relates to #49091 Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

This commit adds the "_async_searhc" get and delete APIs to the AsyncSearchClient in the High Level Rest Client. Relates to #49091

This commit adds the "_async_searhc" get and delete APIs to the AsyncSearchClient in the High Level Rest Client. Relates to elastic#49091 Backport of elastic#53828

This commit adds the "_async_searhc" get and delete APIs to the AsyncSearchClient in the High Level Rest Client. Relates to #49091 Backport of #53828

jimczi · 2020-04-08T17:48:54Z

All the blockers have been merge in 7.7, hence closing. We'll create new issues for improvements related to async search in the future.

javanna added :Search/Search Search-related issues that do not fall into other categories Meta labels Nov 14, 2019

javanna assigned javanna and jimczi Nov 14, 2019

jimczi mentioned this issue Nov 14, 2019

Pre-sort shards based on the max/min value of the primary sort field #49092

Merged

jimczi mentioned this issue Nov 21, 2019

Add a listener to track the progress of a search request locally #49471

Merged

colings86 mentioned this issue Nov 27, 2019

Make EQL REST Querying API async #49638

Closed

jimczi mentioned this issue Dec 6, 2019

Add new x-pack endpoints to track the progress of a search asynchronously #49931

Merged

mchopda added the Dependency:SIEM label Jan 31, 2020

This was referenced Feb 3, 2020

[meta] 7.6 release elastic/elasticsearch-net#4340

Closed

[meta] 7.6 release elastic/elasticsearch-net#4341

Closed

jimczi added blocker v7.7.0 labels Mar 10, 2020

cbuescher mentioned this issue Mar 16, 2020

Add async_search.submit to HLRC #53592

Merged

javanna added a commit to javanna/elasticsearch that referenced this issue Mar 17, 2020

[DOCS] add docs for async search

5a7adf0

Relates to elastic#49091

javanna mentioned this issue Mar 17, 2020

[DOCS] add docs for async search #53675

Merged

cbuescher mentioned this issue Mar 19, 2020

Add async_search get and delete APIs to HLRC #53828

Merged

cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Mar 19, 2020

Add async_search get and delete APIs to HLRC

e98a619

This commit adds the "_async_searhc" get and delete APIs to the AsyncSearchClient in the High Level Rest Client. Relates to elastic#49091

cbuescher mentioned this issue Mar 20, 2020

Add async_search.submit to HLRC #53592 #53852

Merged

javanna added a commit that referenced this issue Mar 20, 2020

[DOCS] add docs for async search (#53675)

0a93a93

Relates to #49091 Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

javanna added a commit that referenced this issue Mar 20, 2020

[DOCS] add docs for async search (#53675)

03fca61

Relates to #49091 Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

cbuescher pushed a commit that referenced this issue Mar 23, 2020

Add async_search get and delete APIs to HLRC (#53828)

3ceb60b

This commit adds the "_async_searhc" get and delete APIs to the AsyncSearchClient in the High Level Rest Client. Relates to #49091

cbuescher mentioned this issue Mar 23, 2020

Add async_search get and delete APIs to HLRC (#53828) #53980

Merged

cbuescher pushed a commit that referenced this issue Mar 23, 2020

Add async_search get and delete APIs to HLRC (#53828) (#53980)

286c366

This commit adds the "_async_searhc" get and delete APIs to the AsyncSearchClient in the High Level Rest Client. Relates to #49091 Backport of #53828

bpintea added v7.8.0 and removed v7.7.0 labels Mar 25, 2020

codebrain mentioned this issue Apr 1, 2020

7.7.0 meta ticket elastic/elasticsearch-net#4525

Closed

38 tasks

jimczi removed the blocker label Apr 8, 2020

jimczi closed this as completed Apr 8, 2020

pugnascotia added v7.7.0 and removed v7.8.0 labels May 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Async search #49091

Async search #49091

javanna commented Nov 14, 2019 •

edited by jimczi

Loading

elasticmachine commented Nov 14, 2019

jimczi commented Apr 8, 2020

Async search #49091

Async search #49091

Comments

javanna commented Nov 14, 2019 • edited by jimczi Loading

elasticmachine commented Nov 14, 2019

jimczi commented Apr 8, 2020

javanna commented Nov 14, 2019 •

edited by jimczi

Loading