StoreGateway not picking external labels in querier metadata API endpoint - /api/v1/label/<label_name>/values

[PavanVasist]

Components it touches - querier, storegateway, sidecar.

Thanos, Prometheus and Golang version used:

Thanos

Version	0.13.0
adf6fac
HEAD
circleci@b8cd18f8b553
20200622-10:04:50
go1.14.2

**Prometheus **

Version	2.14.0
edeb7a44cbf745f1d8be4ea6f215e79e651bfe19
HEAD
root@df2327081015
20191111-14:27:12
go1.13.4

Thanos - quay.io/thanos/thanos:v0.13.0; Prometheus - quay.io/prometheus/prometheus:v2.14.0

Object Storage Provider:
AWS S3 Bucket

What happened:
When external_labels are added to scrape config and tried to filter the label values via metadata API within querier by fetching TSDB from gateway connected to s3 bucket, no values are returned. But when the querier is set directly set to fetch from sidecar, the external_labels are fetched correctly. Some more info -
grafana/grafana#26764
https://groups.google.com/forum/#!msg/prometheus-users/c6kaSHiP7Cc/fILFnptyAQAJ

What you expected to happen:
The label_values for external_labels should be filtered automatically in the querier API metadata endpoint while fetching it from stoegateway.

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:

Environment:

• OS (e.g. from /etc/os-release): Linux/K8s Platform

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[PavanVasist]

Any help or work arounds?

[bwplotka]

Hey, sorry for delay. What's the use case? And I think it makes total sense to uniform Label values to work similarly between StoreAPI implementations.

[PavanVasist]

Thank you for dropping by @bwplotka.
The idea is -

• We have a prometheus and thanos-sidecar running in a cluster pushing metrics to AWS S3 bucket.

• In an another cluster am using thanos-store connected to same S3, with compactor and querier(used as datasource to Grafana).

• Now the prometheus running cluster is short-lived as we run some tests and want to capture the metrics every-time the tests run and later we shutdown the cluster. For every run we add different external_labels to perform a comparison of results across these multiple runs in 2nd cluster where store along with querier is run.

• While defining dashboards in Grafana, we wanted to use external_labels as variable so that the user can choose these and perform analysis.

• But what we observed is, when sidecar is linked to querier as store then we see the external_labels being exposed in metadata API of querier. But when only store-gateway is added as store to querier then metadata API doesn't filter out the values.

Can you please help us out as to what are we missing here.