request: a way to block expensive/problematic queries

[Dieterbe]

Is your feature request related to a problem? Please describe.

Some user created an auto-refreshing grafana dashboard that would periodically call "/tags/autoComplete/values" with a very expensive query (one that would return a large result set out of a large tag index that's slow to search into, I imagine). the http.tagdb-default-limit value doesn't help much because it only filters after the results are generated, and also the query from grafana has this value set to 10k, something the MT admin can't override. (not that restarting a MT cluster to deploy a config update would be feasible). This query is so expensive that many other queries would fail or become very slow. (I don't know the exact specifics on this, there's some backstory here that @replay and @robert-milan are more familiar with)

Describe the solution you'd like

A rejection list for queries. A way to provide a combination a url path and set of parameters that, when found, should be flat out rejected. This way an operator could easily manually intervene by posting an update to the rejection list.

Describe alternatives you've considered

• Automatically block "slow" queries -> to tricky too implement. how do you measure what is slow in a reliable, consistent and fairi way.
• provide more tunables for each specific http path -> lots more hassle and hard to get a similar level of blocking.

[Dieterbe]

added sprint 14 to open the discussion.

[shanson7]

Might be related to this (WIP, PoC) PR too: #1847. Long read requests cause lots of blocking in the index. We have been running the above PR for a while and things are considerably better (although there is still a scenario in which long reads can block things up).

[Dieterbe]

consensus:

• will go in query nodes
• use config file use literal string matching (not regex)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.