How to configure querying data form Long Term Storage

[mdvaliyullah]

what are the configuration required for querying data form LongTerm Storage in grafana

[pracucci]

Mimir queries metrics in the long-term storage by default. Metrics data up to 12h old is required from ingesters, while older data is queries from the long-term storage (object storage) via the Mimir store-gateway component.

Does this answer your question? If not, could you share more details, please?

[mdvaliyullah]

I am setting up mimir locally in monolithic mode connecting to blob storage, I am able to send the data to long term storage but unable to see the data in grafana. below is my configuration details.. am i missing something?

limits:
  enforce_metadata_metric_name: true
common:
  storage:
    backend: ..* #(you can ignore the config for storage)
     
blocks_storage:
  storage_prefix: blocks
  bucket_store:
    sync_dir: /tmp/mimir/tsdb-sync
    sync_interval: 15m
  tsdb:
    dir: /tmp/mimir/tsdb
    flush_blocks_on_shutdown: true
    ship_interval: 1m

frontend:
  align_queries_with_step: true
  log_queries_longer_than: 10s
  split_queries_by_interval: 24h
  cache_results: true
  results_cache:
    backend: "memcached"
    memcached:
      addresses: "127.0.0.1:11211"    

compactor:
  data_dir: /tmp/mimir/compactor
  sharding_ring:
    kvstore:
      store: memberlist
  cleanup_interval: 15m    

distributor:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist

ingester:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
    replication_factor: 1

server:
  http_listen_port: 9009
  log_level: info

store_gateway:
  sharding_ring:
    replication_factor: 1

[pracucci]

The configuration looks good to me. Few questions:

• Are you able to query the last 12h of data (which is queried from ingesters and not from the long-term storage)?
• Do you see any error in the logs?
• Mimir has multi-tenancy enabled by default. Since you mentioned metrics are ingested successfully, I assume you're specifying the X-Scope-OrgID header in the remote write config (Prometheus or Grafana Agent). Are you also configuring the same header in the Grafana data source?

[mdvaliyullah]

I am able query from ingesters but I see some extra datapoint with the same value repeated for 5 mins. attached screen shot for the same. is there any configuration changes required? for information I am pushing the data using mimir-proxy graphite/metrics to mimir

[pracucci]

I am able query from ingesters but I see some extra datapoint with the same value repeated for 5 mins.

This is the PromQL behaviour on stale series. Basically, if no new samples are ingested for a given series (and the series has not been explicitly marked as stale, which is a feature of Prometheus/Agent when scraping targets) then PromQL consider that series stale for the next 5 minutes and the last known data point is used.

[mdvaliyullah]

Also getting below message in the log

level=warn ts=2022-09-22T17:47:07.361030176Z caller=loader.go:122 msg="bucket index not found" user=anonymous

[pracucci]

The bucket index is a per-tenant file (stored in the object storage) which is written by the compactor every cleanup_interval: 15m. Do you see any error in the logs, related to writing / updating the bucket index file?

[mdvaliyullah]

I don't see any errors, but once i run docker compose down and bring up the container. I am not able to see the metrics in the grafana. I was under assumption the data should be fetched from the object storage (long term storage).
flush_blocks_on_shutdown = true // True to flush blocks to storage on shutdown which means to object storage right

[pracucci]

flush_blocks_on_shutdown = true

This is not enough to query the long-term storage for the most recent data. Mimir by default doesn't query the long-term storage for any data point more recent than 12 hours, because it expects that data to be available for querying from ingesters.

[mdvaliyullah]

Thanks for the update