EQL _async_search status API

[lizozom]

We need an API similar to the one implemented for the KQL async search #57537 to check the status of running EQL requests.

The request and response formats should be identical.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[elasticmachine]

Pinging @elastic/es-ql (Team:QL)

[mayya-sharipova]

There are several ways we can organize an API for the status of EQL search:

1. Use the same API as for non EQL searches:

GET /_async_search/status/<ID>

2. Use the EQL specific API

GET /_eql/search/status/<ID>

@lizozom @jimczi What do you think of each option?
I personally favour 2nd option, as EQL can also store the synchronous requests with keep_on_completion parameter, whose status can be checked, thus using _async_search wording in the status API seems incorrect.

@lizozom Is it important for you to have a single API to check the status regardless whether it is EQL search or not? Or is it important for you to differentiate between EQL searches and other searches?

[lizozom]

@mayya-sharipova I have a preference for a single API.
In the context of checking the status - the strategy used to run them doesn't seem to matter, and it would require additional effort to support this in KIbana.
It would be even more important as we move towards having a bulk status API.

[mayya-sharipova]

Thanks @lizozom.
For the unified status API for searches, @jimczi what do you think of these options?

GET _searches_status/<ID>

or

GET _stored_searches_status/<ID>

[jimczi]

I am a bit on the fence because that would mean that we need to differentiate EQL from async_search simply looking at the ID. I can see why it would be useful to have a single API but I am afraid that it complicates things going forward since async searches and EQL behave differently. @lizozom is it really challenging to make the distinction in Kibana ? I guess it would be easier to access this infos in the Kibana Search Sessions than tweaking the ID when we create it.

[lizozom]

@mayya-sharipova after talking to @jimczi, it's fine to have two separate APIs with identical req\res formats.
(And if and when we decide to have a bulk API, we'd pass in the async ids with a type, so that you could work with them more easily)

[mayya-sharipova]

@lizozom @jimczi Thank you, I will work on this.

So for EQL status API, the request format will be:

GET /_eql/search/status/<ID>

The response will be identical to async search status response.

[mayya-sharipova]

I've just learned that eql search response doesn't have information about shards and status. So eql status response will have a different format from async search status responses, like the following format:

{
  "id" : <id>,
  "is_running" : true,
  "is_partial" : true,
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986,
  "timed_out" : false
}

I hope you are fine with this, @lizozom

[lizozom]

@mayya-sharipova this is ok, but not optimal.
This means we won't be able to have any indicator on the progress of the query.
We were planning to show some kind of progress indication using the ration between the successful and total shards.
Is there any alternative indicator we could get for EQL?