Feature/querysharding ii

[owen-d]

What

Following on the tail of #1846, this PR constitutes the stitching & evaluation of sharded queries. For example,

sum(rate({foo="bar"} |="id=123" [1m])) ->

sum(
  downstream<sum(rate(({foo="bar"}|="id=123")[1m])), shard=0_of_2> ++ 
  downstream<sum(rate(({foo="bar"}|="id=123")[1m])), shard=1_of_2>
)

However, this also includes filter queries, like:

{foo="bar"} |= "bazz" -> 
downstream<{foo="bar"}|="bazz", shard=0_of_2> ++ downstream<{foo="bar"}|="bazz", shard=1_of_2>

Support

The current list of shardable optimizations include:

Vector Aggregation Expressions

// sum(x) -> sum(sum(x, shard=1) ++ sum(x, shard=2)...)
// avg(x) -> sum(x)/count(x)
// count(x) -> sum(count(x, shard=1) ++ count(x, shard=2)...)

Range Aggregation Expressions

Both rate and count_over_time are supported.

Why

This allows loki to further parallelise queries. Importantly, it also helps us alleviate parallelization bottlenecks in split factor alone, because with small split factors, multiple queriers inevitably end up downloading the same sets of chunks. Sharding allows us to partition by a dimension other than time, allowing us to combine the two to achieve higher parallelizations with less work duplication.

How

In order to do this without copying some of the messier internals of cortexproject/cortex#1878, this PR extends the api to support a shards field which limits a query to one or more shard values determined by storage. This is only used between the query-frontend & queriers when querier.parallelise-shardable-queries is enabled on the frontend.

[codecov-commenter]

Codecov Report

Merging #1927 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1927   +/-   ##
=======================================
  Coverage   61.58%   61.58%           
=======================================
  Files         149      149           
  Lines       11903    11903           
=======================================
  Hits         7331     7331           
  Misses       3989     3989           
  Partials      583      583           

[cyriltovena]

LGTM !

couple of test missing but nothing majors.

Awesome work !