Added per-tenant in-process sharding support to compactor

[pracucci]

NOTE: Rolled back by #2628

What this PR does:
We're hitting some vertical scalability limits on the compactor. We have a large user with 30+M active series and compacting 2h blocks take more than 2h. The TSDB compactor uses a single CPU core (there's no parallelisation), so we can't really vertically scale up unless sharding blocks.

In this PR I'm introducing per-tenant in-process sharding support to the compactor, leveraging on the fact that Thanos compactor can parallelise compaction of different blocks groups.

The way it works is quite simple:

1. Add the ingester ID as external label to blocks uploaded by the ingester
2. Add a new metadata fetcher filter in the compactor which replaces the ingester ID label with a shard ID calculated on the hash of the ingester ID

Guaranteed properties:

• Same ingester blocks are always compacted together, even for multiple levels of compaction (2nd, 3rd, ... level)

Out of the scope of this PR:

• Per-tenant config overrides

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

Amazing PR!

[pracucci]

Unfortunately this causes problems with the deduplication on the read path, because the new external labels end up being treated as additional series labels. I'm working on a solution.

[pracucci]

Unfortunately this causes problems with the deduplication on the read path, because the new external labels end up being treated as additional series labels. I'm working on a solution.

The solution I've adopted (commit) is pretty simple and follows what also Thanos is doing. The idea is to remove any external label used to identify replicas/shards directly when iterating the bucket.

[pracucci]

Unfortunately this causes problems with the deduplication on the read path, because the new external labels end up being treated as additional series labels. I'm working on a solution.

A quick explanation of the problem and solution.

The Thanos BucketStore keeps blocks grouped by external labels. When such blocks are queried, the external labels are added to the returned Series().

Before this PR we had only the user ID as external label, which is constant across all blocks of an user, so doesn't matter at which "point in time" you remove such external label, we've just to remove it.

However, the new external labels (ingester ID and shard ID) are variable. For the same user we have blocks with different external labels. Think about 2 single series metric_a{} and metric_b{} which are replicated 3 ways and goes to 3 ingesters. When we query it from BucketStore (before compaction) we'll fetch it from 3 blocks (1 per ingester), so we'll get:

metric_a{__ingester_id__="1"}
metric_b{__ingester_id__="1"}
metric_a{__ingester_id__="2"}
metric_b{__ingester_id__="2"}
metric_a{__ingester_id__="3"}
metric_b{__ingester_id__="3"}

In the querier, we assume that a Series() response contains series sorted by label (correct) and thus we assume that all chunks for 1 series are consecutive in the response (in order for the deduplication to work correctly). However, given we now have variable external labels this is no more true and thus the deduplication doesn't work as expected.

The solution (which is also what Thanos does) is removing these external labels that we don't want at query time directly in the metadata fetcher.