Add limit for store gateway downloaded bytes

[yeya24]

What this PR does:

Add the limit from thanos-io/thanos#5801 to Cortex. This is a store gateway only limit that limits all data downloaded for each gRPC request to store gateway.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[alvinlin123]

He @yeya24 do we really need this in addition to max_fetched_data_bytes_per_query? I am trying to understand how max_downloaded_bytes_per_request in store-gateway would work with other Cortex limits related to size of query.

[yeya24]

It is kind of tricky. How Store Gateway works is that based on the matchers of user query, it will try to fetch postings either from cache or from S3.
Users can write very high cardinality matchers to match several GB postings but end up matching no or few series. In this case max_fetched_data_bytes_per_query won't help since it only considers series and chunks bytes that match the result.

I think this limit can cover some edge cases but I am also not sure if it is good to have that. As we don't have the visibility for that and it is enforced per gRPC request, not per query.

[harry671003]

I think this limit can cover some edge cases but I am also not sure if it is good to have that. As we don't have the visibility for that and it is enforced per gRPC request, not per query.

I think this limit is still useful to protect store-gateways from getting OOM killed. We enforce the max_fetched_data_bytes_per_query at the query level in the queriers. The max_downloaded_bytes_per_request should be set equal to the max_fetched_data_bytes_per_query limit.

[alanprot]

LGTM!

In the future we may consider deprecating the other ones? Did it help to prevent OOM?

[yeya24]

In the future we may consider deprecating the other ones? Did it help to prevent OOM?

What are the other ones you mentioned here? The series and chunk limit on store gateway?

[yeya24]

Merging it now.