Sketch: how to stream sorted store responses with dropped replica labels

[MichaHoffmann]

So i was thinking about #6706. We generally want sorted ( not necessarily deduplicated, that happens in proxy store anyhow ) responses from downstream ( from proxy store ) Store APIs.
The usual issue is that some replica labels are inner labels and we cannot guarantee sorted responses with just dropping them. #6706 attempts to solve that by sorting in Store APIs which is not optimal since we need to buffer responses to sort them after the fact.

We can achieve streamed and sorted store responses in the following manner though:

For tsdb store and bucket store:

• fetch label values for replica labels
• fetch postings for replica label values
• fetch postings for all other matchers
• create a respSet for all replica label values by intersecting with the other postings ( note that this means that the replica label value is constant and can be safely dropped without affecting order of the resp Set)
• we now have N respSets that are individually sorted and can be zipped if they are corresponding to proper replica labels but its irrelevant if they are external labels or not which is the nice thing.
• stream the zipped response

For prometheus store:

• we just start streaming noting changes in values of the replica labels; if they change during the stream we return an error since that would mean we deduplicate on inner label which is a user error for sidecar

For proxy store:

• since we can rely on sorted series from downstream store apis we can just use proxy heap + dedup as usual

Like this we could have our cake sorted and stream it too. Does it make sense?

[MichaHoffmann]

I think this is not valuable because generally we need to resort when adding extLabels anyway.