Introduce async search status API #62947

mayya-sharipova · 2020-09-28T11:14:52Z

Introduce async search status API

GET /_async_search/status/

The API is restricted to the monitoring_user role.

Revised version:

For a running async search, the response is:

{
  "id" : <id>,
  "is_running" : true,
  "is_partial" : true,
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986,
  "_shards" : {
      "total" : 562,
      "successful" : 188,
      "skipped" : 0,
      "failed" : 0
  }
}

For a completed async search, an additional
completion_status fields is added.

{
  "id" : <id>,
  "is_running" : false,
  "is_partial" : false,
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986,
  "_shards" : {
      "total" : 562,
      "successful" : 562,
      "skipped" : 0,
      "failed" : 0
  },
 "completion_status" : 200
}

Old version:

For a running async search, the response is:

{
  "id" : <id>,
  "is_running" : true,
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986,
  "_shards" : {
      "total" : 562,
      "successful" : 188,
      "skipped" : 0,
      "failed" : 0
  }
}

For a completed async search, the response is:

{
  "id" : <id>
  "is_running" : false,
  "expiration_time_in_millis" : 1584377890986
}

Techincal details:
We first try to retrieve the status of the async search from tasks.
If this doesn't succeed, we retrieve it from an index: .async-search.
In case of retrieving from the index, we assume that the async search is
completed, and a shorter response for the status is returned.

Closes #57537

elasticmachine · 2020-09-28T11:16:34Z

Pinging @elastic/es-search (:Search/Search)

Introduce async search status API GET /_async_search/status/<id> The API is restricted to the monitoring_user role. For a running async search, the response is: ```js { "id" : <id>, "is_running" : true, "start_time_in_millis" : 1583945890986, "expiration_time_in_millis" : 1584377890986, "_shards" : { "total" : 562, "successful" : 188, "skipped" : 0, "failed" : 0 } } ``` For a completed async search, the response is: ```js { "id" : <id> "is_running" : false, "expiration_time_in_millis" : 1584377890986 } ``` ---- Techincal details: We first try to retrieve the status of the async search from tasks. If this doesn't succeed, we retrieve it from an index: .async-search. In case of retrieving from the index, we assume that the async search is completed, and a shorter response for the status is returned. Closes elastic#57537

mayya-sharipova · 2020-09-28T15:52:13Z

@lizozom I am interested in your feedback on the API above. It returns a more detailed status response for a running async search, and a shorter version for a completed search. Will it suit your needs?

docs/reference/search/async-search.asciidoc

jimczi

Sorry for the late review @mayya-sharipova , I did a first pass and left some comments.

jimczi · 2020-10-20T11:00:59Z

.../plugin/async-search/src/main/java/org/elasticsearch/xpack/search/MutableSearchResponse.java

+     * @param expirationTime – expiration time of async search request
+     * @return response representing the status of async search
+     */
+    AsyncStatusResponse toStatusResponse(String asyncExecutionId, long startTime, long expirationTime) {


This method should be synchronized.

Thanks Jim, good comment. I have made some variables volatile and I believe it doesn't need synchronization any more.

.../plugin/async-search/src/main/java/org/elasticsearch/xpack/search/MutableSearchResponse.java

jimczi · 2020-10-20T11:04:21Z

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/async/AsyncTaskIndexService.java

+     * is registered in the task manager, <code>null</code> otherwise.
+     *
+     */
+    <T extends AsyncTask> T getTaskStatus(TaskManager taskManager, AsyncExecutionId asyncExecutionId, Class<T> tClass) {


Can you add an option on getTask instead to enable/disable the authentication check ?

Thanks for the comment, @jimczi . I have talked to @albertzaharovits about this sometime ago, and he suggested me from a security point of view to use a separate method instead of adding a parameter to enable/disable authentication.

I ended up moving the logic of this method to TransportGetAsyncStatusAction in order to avoid parametrized function arguments.

jimczi · 2020-10-20T11:08:41Z

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/async/AsyncTaskIndexService.java

+     * @param listener – listener to report result to
+     */
+    public void getStatusResponse(
+        AsyncExecutionId asyncExecutionId, BiFunction<String, Long, R> completedStatusProducer, ActionListener<R> listener) {


Can we reuse getEncodedResponse ? We'd need to disable the authentication check but that can be an option on the function directly ?

Similarly to a previous comment, I opted out to have a separate method for this . Please let me know if we still want to have a single method.

...lugin/core/src/main/java/org/elasticsearch/xpack/core/search/action/AsyncStatusResponse.java

docs/reference/search/async-search.asciidoc

mayya-sharipova · 2020-10-20T14:09:03Z

@jimczi Jim, thanks for your review, I will go through it, and address it.

I have another question. It seems that @lizozom and her team are interested in the BULK API for status. I remember we've discussed it, and we came to the conclusion that since we don't have a bulk API for _async_search, we did not want to introduce a bulk API for _async_search/status.
But thinking about this again, it is unlikely that a user ever wants to use a bulk API for _async_search, while a bulk API for _async_search/status seems to be very useful for Kibana.

@jimczi WDYT? Should we reconsider and add a bulk API for _async_search/status?

The API can look like this:

GET /_async_search/status/
{
  "ids" : [<id1>, <id2>]
}

If we decide to go for a bulk version, we can remove a single version where id is provided in the URL.

jimczi · 2020-10-20T15:36:14Z

Should we reconsider and add a bulk API for _async_search/status?

I think that we should first agreed on the metadata that we output for a single status. Once the format is settled, we can discuss whether a bulk format is needed or not but for now I think it's ok to focus on the API to get the status for a single search.

mayya-sharipova · 2020-10-26T18:17:35Z

@lizozom @jimczi Thank you for your feedback, I have tried to address them in the last commit. Please continue the review when you have time

jimczi

The change looks good but I left some comments that need to be addressed.

docs/reference/search/async-search.asciidoc

.../plugin/async-search/src/main/java/org/elasticsearch/xpack/search/MutableSearchResponse.java

jimczi · 2020-10-28T08:43:45Z

...lugin/core/src/main/java/org/elasticsearch/xpack/core/search/action/AsyncStatusResponse.java

+    private final int successfulShards;
+    private final int skippedShards;
+    private final int failedShards;
+    private final RestStatus completionStatus;


I think we can return the status for partial response too ?

...lugin/core/src/main/java/org/elasticsearch/xpack/core/search/action/AsyncStatusResponse.java

...async-search/src/main/java/org/elasticsearch/xpack/search/TransportGetAsyncStatusAction.java

mayya-sharipova · 2020-10-28T19:50:22Z

@jimczi Thanks for another round of feedback. I have tried to address all your comments in 1e427d5 , except the status of a partial search response, which is up for discussion.

jimczi

LGTM, thanks @mayya-sharipova!

jimczi · 2020-10-30T15:48:12Z

.../plugin/async-search/src/main/java/org/elasticsearch/xpack/search/MutableSearchResponse.java

+        if (queryFailures == null) {
+            return 0;
+        }
+        int count = 0;


nit: you can do return queryFailures.asList().size() ?

@jimczi Thanks for the suggestion. It seems that asList() method creates a additional list object. Instead, in 39dafa9, I've added AtomicArray::nonNullLength method.

mayya-sharipova · 2020-11-02T13:13:12Z

@lizozom I am wondering if you are ok with the format of the revised version of status response?

lizozom

LGTM!

Thanks for the revision

Introduce async search status API GET /_async_search/status/<id> The API is restricted to the monitoring_user role. For a running async search, the response is: ```js { "id" : <id>, "is_running" : true, "is_partial" : true, "start_time_in_millis" : 1583945890986, "expiration_time_in_millis" : 1584377890986, "_shards" : { "total" : 562, "successful" : 188, "skipped" : 0, "failed" : 0 } } ``` For a completed async search, an additional `completion_status` fields is added. ```js { "id" : <id>, "is_running" : false, "is_partial" : false, "start_time_in_millis" : 1583945890986, "expiration_time_in_millis" : 1584377890986, "_shards" : { "total" : 562, "successful" : 562, "skipped" : 0, "failed" : 0 }, "completion_status" : 200 } ``` Closes elastic#57537 Backport for elastic#62947

This was missed in elastic#62947 Relates to elastic#62947

Introduce async search status API GET /_async_search/status/<id> The API is restricted to the monitoring_user role. For a running async search, the response is: ```js { "id" : <id>, "is_running" : true, "is_partial" : true, "start_time_in_millis" : 1583945890986, "expiration_time_in_millis" : 1584377890986, "_shards" : { "total" : 562, "successful" : 188, "skipped" : 0, "failed" : 0 } } ``` For a completed async search, an additional `completion_status` fields is added. ```js { "id" : <id>, "is_running" : false, "is_partial" : false, "start_time_in_millis" : 1583945890986, "expiration_time_in_millis" : 1584377890986, "_shards" : { "total" : 562, "successful" : 562, "skipped" : 0, "failed" : 0 }, "completion_status" : 200 } ``` Closes #57537 Backport for #62947

mayya-sharipova added the :Search/Search Search-related issues that do not fall into other categories label Sep 28, 2020

elasticmachine added the Team:Search Meta label for search team label Sep 28, 2020

mayya-sharipova added v8.0.0 v7.10.0 >feature and removed Team:Search Meta label for search team labels Sep 28, 2020

mayya-sharipova force-pushed the async-metadata branch from fb4373c to 5a9b42b Compare September 28, 2020 14:24

mayya-sharipova force-pushed the async-metadata branch from 5a9b42b to faafd18 Compare September 28, 2020 15:50

lizozom reviewed Oct 4, 2020

View reviewed changes

docs/reference/search/async-search.asciidoc Show resolved Hide resolved

andreidan added v7.11.0 and removed v7.10.0 labels Oct 7, 2020

jimczi requested changes Oct 20, 2020

View reviewed changes

mayya-sharipova added 2 commits October 21, 2020 09:37

Merge remote-tracking branch 'upstream/master' into async-metadata

6e38427

Address feedback

0775cf1

jimczi requested changes Oct 28, 2020

View reviewed changes

Address feedback 2

1e427d5

jimczi approved these changes Oct 30, 2020

View reviewed changes

mayya-sharipova added 2 commits November 2, 2020 10:09

Merge remote-tracking branch 'upstream/master' into async-metadata

25c4523

Add nonNullLength to AtomicArray

39dafa9

lizozom approved these changes Nov 3, 2020

View reviewed changes

mayya-sharipova merged commit 074f7d2 into elastic:master Nov 3, 2020

mayya-sharipova deleted the async-metadata branch November 3, 2020 16:35

mayya-sharipova mentioned this pull request Nov 3, 2020

Async search status #64554

Merged

mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this pull request Nov 3, 2020

Add GetAsyncStatusAction to Xpack client actions

ae60071

This was missed in elastic#62947 Relates to elastic#62947

mayya-sharipova mentioned this pull request Nov 3, 2020

Add GetAsyncStatusAction to Xpack client actions #64559

Closed

lizozom mentioned this pull request Dec 6, 2020

Add async_search/status to ESClient #65919

Closed

This was referenced Dec 7, 2020

Add async_search/status API to the JS client elastic/elasticsearch-js#1368

Closed

Async Search Status return expired #66114

Closed

stevejgordon mentioned this pull request Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

pugnascotia changed the title ~~Async search status~~ Introduce async search status API Jan 4, 2021

mayya-sharipova mentioned this pull request Apr 22, 2021

Separe STATUS from RESULT in async search #72115

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

quux00 mentioned this pull request Mar 20, 2024

Allow users with read role to use the async_search/status endpoint #106561

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce async search status API #62947

Introduce async search status API #62947

mayya-sharipova commented Sep 28, 2020 •

edited

Loading

elasticmachine commented Sep 28, 2020

mayya-sharipova commented Sep 28, 2020 •

edited

Loading

jimczi left a comment

jimczi Oct 20, 2020

mayya-sharipova Oct 26, 2020

jimczi Oct 20, 2020

mayya-sharipova Oct 26, 2020

jimczi Oct 20, 2020

mayya-sharipova Oct 26, 2020

mayya-sharipova commented Oct 20, 2020 •

edited

Loading

jimczi commented Oct 20, 2020

mayya-sharipova commented Oct 26, 2020

jimczi left a comment

jimczi Oct 28, 2020

mayya-sharipova commented Oct 28, 2020

jimczi left a comment

jimczi Oct 30, 2020

mayya-sharipova Nov 2, 2020

mayya-sharipova commented Nov 2, 2020

lizozom left a comment

Introduce async search status API #62947

Introduce async search status API #62947

Conversation

mayya-sharipova commented Sep 28, 2020 • edited Loading

Revised version:

Old version:

elasticmachine commented Sep 28, 2020

mayya-sharipova commented Sep 28, 2020 • edited Loading

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayya-sharipova commented Oct 20, 2020 • edited Loading

jimczi commented Oct 20, 2020

mayya-sharipova commented Oct 26, 2020

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayya-sharipova commented Oct 28, 2020

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayya-sharipova commented Nov 2, 2020

lizozom left a comment

Choose a reason for hiding this comment

mayya-sharipova commented Sep 28, 2020 •

edited

Loading

mayya-sharipova commented Sep 28, 2020 •

edited

Loading

mayya-sharipova commented Oct 20, 2020 •

edited

Loading