forked from elastic/elasticsearch
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Relates to elastic#49091
- Loading branch information
Showing
9 changed files
with
248 additions
and
30 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| [role="xpack"] | ||
| [testenv="basic"] | ||
| [[async-search-intro]] | ||
| == Long running searches | ||
|
|
||
| {es} generally allows to quickly search across big amounts of data. There are | ||
| situations where a search executes on many many shards, possibly against | ||
| <<frozen-indices>> and spanning multiple <<modules-remote-clusters>>, for which | ||
| results are not expected to be returned in milliseconds. When needing to | ||
| execute long running searches, sending a search request and synchronously | ||
| waiting for its results to be returned is not ideal. Async Search allows | ||
| instead to submit a search request that gets executed asynchronously and | ||
| monitor its progress as well as retrieve its results at a later stage. | ||
| Also, partial results can be retrieved as they become available, before | ||
| the search has completed. | ||
|
|
||
| An async search request can be submitted using the <<submit-async-search>> API. | ||
| The <<get-async-search>> API allows to monitor the progress of an async search | ||
| request and retrieve its results. An async search can be deleted through the | ||
| <<delete-async-search>> API. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,197 @@ | ||
| [role="xpack"] | ||
| [testenv="basic"] | ||
| [[async-search]] | ||
| === Async Search | ||
|
|
||
| Async search consists of a set of API that allow to asynchronously execute a | ||
| search request, monitor its progress as well as retrieve its partial results | ||
| as soon as they become available. | ||
|
|
||
| [[submit-async-search]] | ||
| ==== Submit Async Search | ||
|
|
||
| Submits a search request that gets executed asynchronously. It accepts the same | ||
| request as the <<search-search,search API>>. | ||
|
|
||
| [source,console,id=submit-async-search-date-histogram-example] | ||
| -------------------------------------------------- | ||
| POST /sales*/_async_search?size=0 | ||
| { | ||
| "sort" : [ | ||
| { "date" : {"order" : "asc"} } | ||
| ], | ||
| "aggs" : { | ||
| "sale_date" : { | ||
| "date_histogram" : { | ||
| "field" : "date", | ||
| "calendar_interval": "1d" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // TEST[setup:sales] | ||
| // TEST[s/size=0/size=0&wait_for_completion=0/] | ||
|
|
||
| The response contains the identifier of the search being executed, to be used | ||
| to retrieve the results at a later stage. Also, the currently available search | ||
| results are returned as part of the <<search-api-response-body,`response`>> object. | ||
|
|
||
| [source,console-result] | ||
| -------------------------------------------------- | ||
| { | ||
| "id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=", <1> | ||
| "version" : 0, | ||
| "is_partial" : true, <2> | ||
| "is_running" : true, <3> | ||
| "start_time_in_millis" : 1583945890986, | ||
| "expiration_time_in_millis" : 1584377890986, | ||
| "response" : { | ||
| "took" : 1122, | ||
| "timed_out" : false, | ||
| "num_reduce_phases" : 0, | ||
| "_shards" : { | ||
| "total" : 562, <4> | ||
| "successful" : 3, <5> | ||
| "skipped" : 0, | ||
| "failed" : 0 | ||
| }, | ||
| "hits" : { | ||
| "total" : { | ||
| "value" : 157483, <6> | ||
| "relation" : "gte" | ||
| }, | ||
| "max_score" : null, | ||
| "hits" : [ ] | ||
| } | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE[s/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=/$body.id/] | ||
| // TESTRESPONSE[s/1583945890986/$body.start_time_in_millis/] | ||
| // TESTRESPONSE[s/1584377890986/$body.expiration_time_in_millis/] | ||
| // TESTRESPONSE[s/"took" : 1122/"took": $body.response.took/] | ||
| // TESTRESPONSE[s/"total" : 562/"total": $body.response._shards.total/] | ||
| // TESTRESPONSE[s/"successful" : 3/"successful": $body.response._shards.successful/] | ||
| // TESTRESPONSE[s/"value" : 157483/"value": $body.response.hits.total.value/] | ||
|
|
||
| <1> Identifier of the async search that can be used to monitor its progress, retrieve its results, and/or delete it. | ||
| <2> Whether the returned search results are partial or final | ||
| <3> Whether the search is still being executed or it has completed | ||
| <4> How many shards the search will be executed on, overall | ||
| <5> How many shards have successfully completed the search | ||
| <6> How many documents are currently matching the query, which belong to the shards that have already completed the search | ||
|
|
||
| It is possible to block and wait until the search is completed up to a certain | ||
| timeout by providing the `wait_for_completion` parameter, which defaults to | ||
| `1` second. | ||
|
|
||
| The Submit Async Search API supports the same <<search-search-api-query-params,parameters>> | ||
| as the search API, though some have different default values: | ||
| `pre_filter_shard_size` defaults to `1`, `batched_reduce_size` defaults to | ||
| `5`, `request_cache` defaults to `true` and `ccs_minimize_roundtrips` | ||
| defaults to `false`. | ||
|
|
||
| It is also possible to specify how long the async search needs to be | ||
| available through the `keep_alive` parameter, which defaults to `5d` (five days). | ||
| Running async searches as well as saved search results are cleared once their | ||
| corresponding `keep_alive` expires. | ||
|
|
||
| WARNING: Async search does not support scroll searches, nor search requests that | ||
| only include the `suggest` section. Cross cluster searches can be executed using | ||
| async search but only with `ccs_minimize_roundtrips` disabled. | ||
|
|
||
| [[get-async-search]] | ||
| ==== Get Async Search | ||
|
|
||
| The Get Async Search API can be used to retrieve the results of a previously | ||
| submitted async search request given its id. | ||
|
|
||
| [source,console,id=get-async-search-date-histogram-example] | ||
| -------------------------------------------------- | ||
| GET /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc= | ||
| -------------------------------------------------- | ||
| // TEST[continued s/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=/\${body.id}/] | ||
|
|
||
| [source,console-result] | ||
| -------------------------------------------------- | ||
| { | ||
| "id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=", | ||
| "version" : 2, <1> | ||
| "is_partial" : true, | ||
| "is_running" : true, | ||
| "start_time_in_millis" : 1583945890986, | ||
| "expiration_time_in_millis" : 1584377890986, | ||
| "response" : { | ||
| "took" : 12144, | ||
| "timed_out" : false, | ||
| "num_reduce_phases" : 38, | ||
| "_shards" : { | ||
| "total" : 562, | ||
| "successful" : 188, | ||
| "skipped" : 0, | ||
| "failed" : 0 | ||
| }, | ||
| "hits" : { | ||
| "total" : { | ||
| "value" : 456433, | ||
| "relation" : "eq" | ||
| }, | ||
| "max_score" : null, | ||
| "hits" : [ ] | ||
| }, | ||
| "aggregations" : { <2> | ||
| "sale_date" : { | ||
| "buckets" : [] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE[s/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=/$body.id/] | ||
| // TESTRESPONSE[s/"is_partial" : true/"is_partial" : false/] | ||
| // TESTRESPONSE[s/"is_running" : true/"is_running" : false/] | ||
| // TESTRESPONSE[s/1583945890986/$body.start_time_in_millis/] | ||
| // TESTRESPONSE[s/1584377890986/$body.expiration_time_in_millis/] | ||
| // TESTRESPONSE[s/"took" : 12144/"took": $body.response.took/] | ||
| // TESTRESPONSE[s/"total" : 562/"total": $body.response._shards.total/] | ||
| // TESTRESPONSE[s/"successful" : 188/"successful": $body.response._shards.successful/] | ||
| // TESTRESPONSE[s/"value" : 456433/"value": $body.response.hits.total.value/] | ||
| // TESTRESPONSE[s/"buckets" : \[\]/"buckets": $body.response.aggregations.sale_date.buckets/] | ||
| // TESTRESPONSE[s/"num_reduce_phases" : 38,//] | ||
|
|
||
| <1> The returned `version` is useful to identify whether the response contains | ||
| additional results compared to previously obtained responses. If the version | ||
| stays the same, no new results have become available, otherwise a higher version | ||
| number indicates that more shards have completed their execution of the query | ||
| and their partial results are also included in the response. | ||
| <2> Partial aggregations results, coming from the shards that have already | ||
| completed the execution of the query. | ||
|
|
||
| NOTE: When results are sorted by a numeric field, shards get sorted based on | ||
| minimum and maximum value that they hold for that field, hence partial | ||
| results become available following the sort criteria that was requested. | ||
|
|
||
| The `wait_for_completion` parameter, which defaults to `1`, can also be provided | ||
| when calling the Get Async Search API, in order to wait for the search to be | ||
| completed up until the provided timeout. Final results will be returned if | ||
| available before the timeout expires, otherwise the currently available results | ||
| will be returned once the timeout expires. | ||
|
|
||
| The `keep_alive` parameter, which defaults to `5d` (five days) can be provided | ||
| to specify how long the async search should be available in the cluster. Once | ||
| an async search expires, it is cancelled if still running and its saved results | ||
| are deleted if already completed. | ||
|
|
||
| [[delete-async-search]] | ||
| ==== Delete Async Search | ||
|
|
||
| The Delete Async Search API can be used to manually delete a specific async | ||
| search given its id. The corresponding async search will be cancelled if it's | ||
| still running, otherwise its saved results will be deleted. | ||
|
|
||
| [source,console,id=delete-async-search-date-histogram-example] | ||
| -------------------------------------------------- | ||
| DELETE /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc= | ||
| -------------------------------------------------- | ||
| // TEST[continued s/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=/\${body.id}/] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.