query frontend: sorted queries do not produce sorted results for shardable queries

[sdufel]

Thanos, Prometheus and Golang version used:
thanos, version (branch: , revision: 2d6b0d4)
build user:
build date:
go version: go1.19
platform: darwin/arm64
(Latest main as of 2023-01-19)

Object Storage Provider: N/A

What happened:
When running a query-frontend with vertical sharding enabled, wrapping shardable instant queries with sort() or sort_desc() does not
produce sorted results.

What you expected to happen:
Instant queries wrapped in a sort() should return results ordered by value

How to reproduce it (as minimally and precisely as possible):
Run a query-frontend with --query-frontend.vertical-shards=2
Issue a shardable query wrapped with sort, e,g,:

sort(sum(rate(container_cpu_usage_seconds_total{}[2m])) by (pod))

Full logs to relevant components:
N/A

[yeya24]

This seems a bug indeed. I can see if sort or sort_desc is the root expression, then we will hit the bug because we always sort by series labels when merging, not by sample values.
We can:

1. disable sharding if sort or sort_desc is in the root expression
2. Sort based on values if sort or sort_desc is in the root expression

[fpetkovski]

We should be able to implement 2. fairly easily. We did something similar in the engine for topk/bottomk: https://github.com/thanos-community/promql-engine/pull/156/files

[alanprot]

I think in the topk/bottomk is ok to sort at the and as when using those functions probably we will not be returning lots of results. I'm a bit more worried on the general case though: I can have a sort that returns lots of series and we will now have to sort it again on the QF. I think we should at least do a k-way merge there as the partitioned results will be already sorted.