Chunk and index retention per tenant

[aylei]

Does loki support chunk and index retention? I'd like to configure the retention time of index and chunks instead of deleting chunks and update the index manually.

And it will be better if retention time could be configured per tenant, which empowers loki as a logging-as-a-service.

[gouthamve]

Yes, the plan to have retention per tenant. How to implement efficiently needs to be seen. If all tenants have the same retention, deleting old data would be as simple as deleting a table and a bucket once it becomes too old (we cut a new table and bucket every week).

But for retention per tenant, we might need to do Batch deletes which might be costly and take time. It needs to be seen how to do it efficiently. But the good thing is all that the data for each tenant is prefixed with the tenantID.

[aylei]

@gouthamve Thanks for the quick and informative response. Despite the efficiency of chunks deleting, which component of loki will actually do the deleting job?

[gouthamve]

I think it should be part of the table manager (its is a Cortex component that manages the creation of the weekly tables).

[dguendisch]

Btw, would be great if the planned retention feature could support time-based and volume-based retention at the same time.
A time-based retention (e.g. two weeks) is what I'd use for data privacy reasons, a volume-based is what lets me sleep well as I don't have to fear that loki breaks once a volume becomes full (I'd only loose e.g. some old logs and instead of 14 days of logs I then have temporarily only 12 days of logs, I can then resize the volume during business hours).
If that would work together at the same time, that would be great.

[tklovett]

A time-based log retention policy may be a requirement for use of Loki under the new CCPA regulations. For example, companies may need a strong guarantee that NGINX or application logs containing PII (e.g. UserIDs, I think) will be deleted after 45 / 90 days.

The bill would grant a consumer the right to request deletion of personal information and would require the business to delete upon receipt of a verified request, as specified.
...
(1) A time period for a business to respond to any verified consumer request may be extended by up to 90 additional days where necessary, taking into account the complexity and number of the requests. The business shall inform the consumer of any such extension within 45 days of receipt of the request, together with the reasons for the delay.

- From the The California Consumer Privacy Act of 2018

[thomasf]

A time-based log retention policy may be a requirement for use of Loki under the new CCPA regulations.

And the GDPR requires us to be able to purge information regarding speicfic users so we need a way to rewrite logs, maybe even through atomic batch jobs of some sort where all lines are filtered through code if regular expressions aren't enough.

[svparijs]

Whats to status of these wishes/requirements? We'd really like to start looking into using Loki for our monitoring/logging production setup. But have some serieus requirements in regards to retention rules per log type, GDPR, etc. And if this is not yet in place is there a roadmap?

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[metri]

Please, don't close issue.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[metri]

remove stale

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[hassanfarid]

I am unable to make retention work yet. Is this in progress?

I am using loki:1.0.0 with following configurations,

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 1

schema_config:
  configs:
  - from: 2018-04-15
    store: boltdb
    object_store: filesystem
    schema: v9
    index:
      prefix: index_
      period: 168h

storage_config:
  boltdb:
    directory: /tmp/loki/index

  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0

table_manager:
  retention_deletes_enabled: true
  retention_period: 600000ms

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[sudeep1607]

Is there a plan or road map for this to be implemented?

[cyriltovena]

Yeah this is going to happen first in Cortex and then Loki.

[midnightconman]

Can we have an update on this, please? Also, please link any related cortex issues / PRs.

Thank you! 😸

[rbel-io]

Isn't the requested feature already implemented?

I would like to see a size-based (in disk storage space) retention rule as well so we could use the following logic:

If the time retention rule or the size retention rule is met, cleanup the Chunk+Index per tenant.

[kubicgruenfeld]

Cortex got per tenant retention through compactor implemented recently. Not sure if it can be useful in Loki as well the way it is.
cortexproject/cortex#3879