Improve ingester reads

[cyriltovena]

When running queries across high throughput streams, we can see that deduplications from ingester data in querier is hurting a lot and time consuming.

For example one query ran for 38s with those logs as information:

Fetched chunks | 264 |  
-- | -- | --
Time Fetching chunk (ms) | 954 |  
Total Duplicates | 16322155 |  
Total bytes compressed (MB) | 150 |  
Total bytes uncompressed (MB) | 980 |  
Total exec time (ms) | 38000 |  
level | "debug"

As you can see, ingesters seems to send tons of duplicates, causing slow down.

Currently we query ingesters for the whole time range of the request.

I propose that we find the latest chunk time for each streams in the storage, and use that as part of the query for the ingesters, minimising result sent from ingesters to only what we don't have from the storage.

With this map of metric name to start time, we should be able to build a different stream iterator here
https://github.com/grafana/loki/blob/master/pkg/ingester/instance.go#L203 that would have a better start time.

/cc @gouthamve @slim-bean @owen-d

Although it should be noted that this might not improve performance by a lot since we run with:

  chunk_idle_period: 15m
  chunk_retain_period: 1m

[owen-d]

Cortex handles this via a querier.query-ingesters-within configuration (which can be set to ingester.max-chunk-age. That sounds like a good place to start as it's a low lift which may help us validate if the strategy you're describing is necessary.

[cyriltovena]

Just looked at our p99 for age chunks and it's around 5 to 15 hours, so we do have a lot of chunks in memory.

Apparently cortex had this issue and fixed it by batching ingester reads.

[slim-bean]

@cyriltovena I believe we could close this now? We have fixed chunks lengths now as well as query-ingesters-within WDYT?

[slim-bean]

Closing