Protection against OOMs in query-frontend?

[sebhoss]

We have one insane Grafana dashboard that executes not just an absurd amount of queries upon loading, each of those queries return a large set of data as well which routinely causes replicas of query-frontend to OOM in our setup. There is of course a political aspect here which prohibits me from changing anything in that dashboard and we do run multiple replicas anyway so an OOM every now and then does not really cause an issue for us... BUT I would really love to not see our replicas to crash.

So I'm wondering if there is any setting or something else that I could do to protect against OOMs in query-frontend. Any idea/suggestions highly appreciated!