Expose query.max-samples flag for Thanos Querier.

[bwplotka]

Thanks of upgrade of promQL package, we got new Prometheus change:

[FEATURE] Make maximum number of samples per query configurable via --query.max-samples flag. #4513

Which gives ability to limit number of samples. This with max-concurrent queries, allows to calculate maximum amount of samples Querier can hold and set it accordingly.

Open question: Does max-samples helps with blocking OOMs of StoreAPI? At first glance, yes as we do max sample verify while we are iterating over messages from stream, so we will not fetch more much more samples than query.max-samples define. (However store API implementation can buffer more e.g sidecar with non-streamed remote-read).

Acceptance Criteria:

• expose query.max-samples for Thanos Querier
• Define default value. Should be unlimited (IMO. no)

CC @domgreen

[GiedriusS]

@bwplotka does this even make sense? Could you PTAL at this comment here: #1369 (comment)?

[ppanyukov]

Open question: Does max-samples helps with blocking OOMs of StoreAPI?

Short answer: no. Or more precisely, "it depends".

From my own tests:

• this is a difficult limit to use meaningfully;
• the actual memory allocated by PromQL very largely depends on the query and the cardinality;
• so querier can still OOM.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[fpetkovski]

@GiedriusS do you think it makes sense to allow configuring this (or a similar limit) in the lazySeriesSet?

[GiedriusS]

I think it's a bit too late there. I think the limit should apply before Recv() https://github.com/thanos-io/thanos/blob/main/pkg/store/proxy.go#L506