Query Deduplication Only Removes Labels

[dh1656]

I am attempting to configure a new Prometheus/Thanos HA configuration that is running on 2 servers. The installation is running on RHEL 9 each running both Prometheus and Thanos components. The pair is configured to scrape a custom API endpoint reading NSX metrics.

Thanos, Prometheus and Golang version used:
Thanos: 0.30.2 and 0.31.0
GoLang: go1.19.7
Prometheus: 2.44

$ thanos --version
thanos, version 0.31.0 (branch: HEAD, revision: 50c4641)
build user: root@63f5f37ee4e8
build date: 20230323-10:13:38
go version: go1.19.7
platform: linux/amd64

Object Storage Provider:
Internal S3 Provider

What happened:
When looking for data via query, selecting Use Deduplication, the external labels configured in Prometheus and query disappear, but all time series remain.

What you expected to happen:
I would expect to see only a single time series returned. As an example, we are returning an integer of 0-2 to show cluster health. We want query to show a single value from one of the cluster members, but show the same value from both sources.

How to reproduce it (as minimally and precisely as possible):
Prometheus ex
NSX_Cluster_Status{job="jobname", environment="production"}

Full logs to relevant components:
Please let me know what logs are needed.

Anything else we need to know:
Prometheus configs:
Server 1
global:
scrape_interval: 5s
evaluation_interval: 15s
external_labels:
cluster: "dc1"
replica: "prometheus-1"
environment: "production"

Server 2
global:
scrape_interval: 5s
evaluation_interval: 15s
external_labels:
cluster: "dc1"
replica: "prometheus-2"
environment: "production"

Thanos Query Service
[Unit]
Description=Thnaos Query
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/bin/thanos query
--http-address=0.0.0.0:29090
--grpc-address=0.0.0.0:10903
--endpoint=prometheus-1:10901
--endpoint=prometheus-2:10901
--query.replica-label=replica \

[Install]
WantedBy=multi-user.target

[douglascamata]

@dh1656 Thanos v0.31 has a deduplication bug. You should use v0.30.2.

You put both versions in your report, so I'm confused, because we aren't aware of any deduplication bug with v0.30.2.

[sjentzsch]

To add to it: Should be a duplicate of #6257