Different Query Result about Recording Rules and Simple Time Series in Thanos Query

[debuggedme]

I have three Prometheus instances with sharding, collecting some time series. Some time series are generated through recording rules, while others are simple scraped metrics. In Thanos Query, the same behavior for both types of time series yields different results.

For example, the simple scraped metric up without using "Use Deduplication" results in:

sum(up) by (prometheus_replica)
---
{prometheus_replica="prometheus-kp-prometheus-shard-2-0"} 1
{prometheus_replica="prometheus-kp-prometheus-shard-2-1"} 1
{prometheus_replica="prometheus-kp-prometheus-shard-1-0"} 1
{prometheus_replica="prometheus-kp-prometheus-shard-1-1"} 1
{prometheus_replica="prometheus-kp-prometheus-0"} 1
{prometheus_replica="prometheus-kp-prometheus-1"} 1

sum(up)
---
{} 6

Using "Use Deduplication" gives:

sum(up) by (prometheus_replica)
---
{} 3

sum(up)
---
{} 3

For the recording rule generated time series up:sum, without deduplication, the result is:

sum(up:sum) by (prometheus_replica)
---
{prometheus_replica="prometheus-kp-prometheus-shard-2-0"} 1
{prometheus_replica="prometheus-kp-prometheus-shard-2-1"} 1
{prometheus_replica="prometheus-kp-prometheus-shard-1-0"} 1
{prometheus_replica="prometheus-kp-prometheus-shard-1-1"} 1
{prometheus_replica="prometheus-kp-prometheus-0"} 1
{prometheus_replica="prometheus-kp-prometheus-1"} 1

sum(up:sum)
---
{} 6

With deduplication, the result is:

sum(up:sum) by (prometheus_replica)
---
{} 1

sum(up:sum)
---
{} 1

I think that both recording-generated time series and ordinary time series should appear the same in Thanos, so the results should not differ.

If my understanding is incorrect, please point it out.

Here are some relevant configurations from the Bitnami-deployed Prometheus:

global:
  external_labels:
    cluster: A
    prometheus: monitoring/kp-prometheus
    prometheus_replica: prometheus-kp-prometheus-1

Here is Thanos Query helm file:

  ## @param query.replicaLabel Replica indicator(s) along which data is de-duplicated
  ##
  replicaLabel:
    - prometheus_replica

Prometheus startup parameters are as follows:

  - args:
    - --web.console.templates=/etc/prometheus/consoles
    - --web.console.libraries=/etc/prometheus/console_libraries
    - --config.file=/etc/prometheus/config_out/prometheus.env.yaml
    - --web.enable-lifecycle
    - --web.enable-remote-write-receiver
    - --web.external-url=http://kp-prometheus.monitoring:9090/ 
    - --web.route-prefix=/
    - --storage.tsdb.retention.time=30d
    - --storage.tsdb.retention.size=2GB
    - --storage.tsdb.path=/prometheus
    - --web.config.file=/etc/prometheus/web_config/web-config.yaml
    - --storage.tsdb.max-block-duration=2h
    - --storage.tsdb.min-block-duration=2h
    image: bitnami/prometheus:2.43.0-debian-11-r2

Thanos Query startup parameters:

  - args:
    - sidecar
    - --prometheus.url=http://localhost:9090
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --tsdb.path=/prometheus/
    - --objstore.config=$(OBJSTORE_CONFIG)
    image: bitnami/thanos:0.31.0-scratch-r0

[debuggedme]

got it