Loki consumes lots of memory even on empty queries with a long time span

[marcinbarczynski]

Describe the bug
The longer the query time span, the more memory is consumed by Loki. It applies even if there are no query results.

To Reproduce

1. Started Loki 1.6.1 and Promtail 1.6.1.
2. Query that is not expected to return any results but has a long time span:

   $ logcli-linux-amd64 query '{filename=~".*my.log"}|="no-chance-it-will-match-anything"' --from="1980-01-01T00:00:00Z" --limit=10
   

   Start date was chosen to include all possible logs.
3. After a few minutes service loki status reported 11 GB of used memory, and the process was killed by OOM killer.
   More reasonable interval of 1 month, RAM usage reported by service loki status is ~7 GB.

Expected behavior
Query succeeds, and the RAM usage is low.

Screenshots, Promtail config, or terminal output

Right before loki process got killed, I had collected metrics: metrics.txt and ran go tool pprof:

Config:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_server_max_concurrent_streams: 1000

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
  - from: 2020-01-01
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h
    chunks:
      prefix: chunk_
      period: 24h

storage_config:
  boltdb:
    directory: /opt/grafana/loki/index

  filesystem:
    directory: /opt/grafana/loki/chunks

limits_config:
  ingestion_rate_mb: 100
  ingestion_burst_size_mb: 200
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 720h
  max_streams_per_user: 0
  max_streams_matchers_per_query: 100
  max_entries_limit_per_query: 10000

table_manager:
  retention_deletes_enabled: true
  retention_period: 720h

[cyriltovena]

This is because you're using this matcher {filename=~".*my.log"} this forces loki to get all entries in the index.

[marcinbarczynski]

Thanks for the quick reply.

So it means that before checking other filters (|="no-chance-it-will-match-anything"' in my case), Loki loads all matching index entries to memory, right?

The number of index entries is correlated with the number of streams, ergo the number of unique label sets. One way to alleviate the problem is to reduce the number of labels.

[cyriltovena]

Indeed, add a single matcher like namespace="dev" or cluster="us-central1" and it should be better.

I'm looking into improving the index allocations, but that's long term I wouldn't wait on it.

[marcinbarczynski]

Thanks.