Store gateway returns no metrics even if blocks loaded

[caymard]

Hi,

I'm facing this issue since several days and did not find any way to understand what is happening.

I have several sidecars that are pushing 2-hours blocks on a bucket, a compactor is doing compaction/downsampling, and the store gateway to get those metrics.

The problem is : when I try to fetch some metrics only from the store gateway with thanos query (with endpoint filter enabled), I got 0 results (with the test query count(up).
Therefor, on Store Gateway UI, I see all my loaded blocks, and with the tool thanos tools bucket inspect also.

Why does the store gateway returns absolutely no results ? How can I found some debug data, more than the debug logs ?

Thanks

[dosubot[bot]]

Hey @caymard! 👋 I'm here to help you out with any bugs, questions, or contributions you might have!

The store gateway returns no results when fetching metrics with Thanos query because of the relabel configuration applied to the store gateway. Specifically, the relabel configuration drops any series with the label ext1 having the value value2.

s1 := e2ethanos.NewStoreGW(
    e,
    "1",
    client.BucketConfig{
        Type:   client.S3,
        Config: e2ethanos.NewS3Config(bucket, m.InternalEndpoint("http"), m.InternalDir()),
    },
    memcachedConfig,
    "",
    nil,
    relabel.Config{
        Action:       relabel.Drop,
        Regex:        relabel.MustNewRegexp("value2"),
        SourceLabels: model.LabelNames{"ext1"},
    },
)

This configuration ensures that any series with the label ext1 set to value2 will be dropped and not returned in the query results ^[1].

Additionally, to identify issues with block loading or querying in the Thanos Store Gateway logs, look for the following specific debug log entries or patterns:

1. Block Overlap Errors:

   • Look for messages indicating critical errors due to block overlaps. These messages typically contain phrases like "critical error detected; halting" and details about the overlapping blocks.
   • Example log entry:

     msg="critical error detected; halting" err="compaction failed: compaction: pre compaction overlap check: overlaps found while gathering blocks. [mint: 1555128000000, maxt: 1555135200000, range: 2h0m0s, blocks: 2]: <ulid: 01D94ZRM050JQK6NDYNVBNR6WQ, mint: 1555128000000, maxt: 1555135200000, range: 2h0m0s>, <ulid: 01D8AQXTF2X914S419TYTD4P5B, mint: 1555128000000, maxt: 1555135200000, range: 2h0m0s>"

2. Connection Refused:

   • Look for warnings about connection issues, which might indicate that Thanos is not properly configured or that Prometheus is not running.
   • Example log entry:

     level=warn ts=2020-04-18T03:07:00.512902927Z caller=intrumentation.go:54 msg="changing probe status" status=not-ready reason="request flags against http://localhost:9090/api/v1/status/config: Get \"http://localhost:9090/api/v1/status/config\": dial tcp 127.0.0.1:9090: connect: connection refused"

3. Missing External Labels:

   • Look for logs indicating that Thanos cannot identify Prometheus due to missing external labels.
   • Example log entry:

     level=info ts=2020-04-18T03:16:32.158536285Z caller=grpc.go:137 service=gRPC/server component=sidecar msg="internal server shutdown" err="no external labels configured on Prometheus server, uniquely identifying external labels must be configured"

These log entries and patterns can help diagnose and troubleshoot issues related to block loading or querying in the Thanos Store Gateway ^[2].

I also found similar discussions that might be relevant to your issue:

1. Recursive reads from object storage subfolders using single Store Gateway ^[3].
2. Is there a way to ask Thanos Store to find a block? ^[4].
3. Unable to get metrics from S3 ^[5].

These discussions might provide additional insights or solutions to your problem.

To continue talking to Dosu, mention @dosu.