API to list running requests

[nik9000]

I searched for but couldn't find a request for an api to list running queries an updates so I'm going to put together my wish list.

1. Only really need queries, indexes, updates, bulk updates, and multi search.
2. Suggest requests would be nice as well but are not required.
3. Should be able to get full request body.
4. The request should have free form fields that I can add to the request so I can recognize requests more easily. I'd put things in there like the ip that caused me to issue the request and the my own identifier for the request type.
5. The requests should have some unique identifier on them that lasts for the life of the request and won't be repeated soon.
6. I don't need to search this stuff though it might help.
7. Needs to return all queries running on the cluster.
8. I won't issue the command frequently but when I do I need it to be serviced even if the cluster is under heavy load. In fact, that is when I'll need it the most.
9. This API is useful even without the ability to kill requests but, one day, killing would be nice.

[ralgara]

I second @nik9000 in his wish list for APIs to list and kill queries. Much needed. The referenced elastic/kibana#1173 and this #4329 reference each other and both are closed. Any light at the end of the tunnel?

[clintongormley]

Today the search context is stored in a thread local. It would be feasible to add an API that would iterate through all of these values (representing currently running searches) and return them, but it would mean changing the data structure to allow locking. We'd also need a way of capturing other types of requests (like bulk etc).

The information would have to be per-node to be of any use.

It might be worth investigating, although I'm leaning towards directing effort towards making changes that prevent abusive requests, things like switching from fielddata to doc values (as we've done in master), improvements to the circuit breakers, and safeguarding users from making silly mistakes (eg #11511).

[nik9000]

It might be worth investigating, although I'm leaning towards directing effort towards making changes that prevent abusive requests, things like switching from fielddata to doc values (as we've done in master), improvements to the circuit breakers, and safeguarding users from making silly mistakes (eg #11511).

This is probably for the best. I wonder if you could do current request listing without something fancy - just keeping a list per node. But meh, just not being able to make bad queries is more likely I think.

The lack of a kill feature is still a big whole when you compare it to relational databases - they always have the feature and I've always used it. Its a bit wish list-y though. PostgreSQL's kill works because each connection has process isolation. It makes locking harder but killing easier. With Elasticsearch you'd have to do something like add Thread.interrupted all over the place or something. Including to Lucene.

[clintongormley]

Closing in favour of #12187