Store gateway hits 422 limit but return 500 at query frontend

[yeya24]

Describe the bug

level=error ts=2023-04-19T08:18:39.873296407Z caller=handler.go:297 msg="query stats" component=query-frontend method=POST path=/prometheus/api/v1/series response_time=26.852163696s query_wall_time_seconds=0 fetched_series_count=0 fetched_chunks_count=0 fetched_samples_count=0 fetched_chunks_bytes=0 fetched_data_bytes=0 status_code=500 param_end=1681892239 param_match[]="{ClusterId=\"test-cluster\",namespace=\"prometheus\"}" param_start=1679300239 error="{\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"failed to receive series from 10.0.35.137:9095: rpc error: code = ResourceExhausted desc = fetch series for block 01GX58AA19BMQZRWBN80PZ2VXG: rpc error: code = ResourceExhausted desc = exceeded series limit: rpc error: code = Code(422) desc = limit 12000 violated (got 12704)\"}"

We saw error above when store gateway hits 422 max series limit error, however query frontend got status code 500 error internal.

This is due to a recent change in Thanos thanos-io/thanos#6171. The gRPC status code got changed to codes.ResourceExhausted and we didn't handle it in Cortex. The gRPC status code got converted to 500 in querier and returned to query frontend.

The same issue could happen for all APIs that touch series or chunks, including instant and range queries, GetSeries, label names and label values API.

[alanprot]

😨

Is this on 1.15? =/

[yeya24]

Yes, it is included.