Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki is deleting all streams rather than per-stream #3881

Closed
atrinh567 opened this issue Jun 22, 2021 · 3 comments
Closed

Loki is deleting all streams rather than per-stream #3881

atrinh567 opened this issue Jun 22, 2021 · 3 comments
Labels
component/compactor feature/retention needs triage Issue requires investigation stale A stale issue or PR that will automatically be closed.

Comments

@atrinh567
Copy link

atrinh567 commented Jun 22, 2021

Describe the bug
Loki is not deleting filtered stream, but all streams. I've pasted my Loki config below that is suppose to apply a retention period to all streams that include that label. Am I missing something?

To Reproduce

    auth_enabled: false

    server:
      http_listen_port: 3100

    distributor:
      ring:
        kvstore:
          store: memberlist

    memberlist:
      join_members:
        - {{ include "loki.fullname" . }}-memberlist

    ingester:
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 1
      chunk_idle_period: 30m
      chunk_block_size: 262144
      chunk_encoding: snappy
      chunk_retain_period: 1m
      max_transfer_retries: 0

    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m
      ingestion_rate_mb: 8
      ingestion_burst_size_mb: 10
      max_query_parallelism: 32
      retention_period: 8760h
      retention_stream:
      - selector: '{app="sample-app"}'
        priority: 1
        period: 24h

    schema_config:
      configs:
        - from: 2021-01-01
          store: boltdb-shipper
          object_store: aws
          schema: v11
          index:
            prefix: loki_index_
            period: 24h
            tags:
              component: loki

    storage_config:
      boltdb_shipper:
        shared_store: aws
        active_index_directory: /var/loki/index
        cache_location: /var/loki/cache
        cache_ttl: 168h

      aws:
        s3: s3://us-east-1/<some-loki-chunk-store-name>

    query_range:
      align_queries_with_step: true
      max_retries: 5
      split_queries_by_interval: 15m
      cache_results: true
      results_cache:
        cache:
          enable_fifocache: true
          fifocache:
            max_size_items: 1024
            validity: 24h

    frontend_worker:
      frontend_address: {{ include "loki.queryFrontendFullname" . }}:9095
      grpc_client_config:
          max_send_msg_size: 1.048576e+08
      parallelism: 6

    frontend:
      log_queries_longer_than: 5s
      compress_responses: true
      tail_proxy_url: http://{{ include "loki.querierFullname" . }}.loki.svc.cluster.local:3100

    compactor:
      working_directory: /var/loki/compactor
      shared_store: aws
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 1m
      retention_delete_worker_count: 150

    ruler:
      storage:
        type: local
        local:
          directory: /etc/loki/rules
      ring:
        kvstore:
          store: memberlist
      rule_path: /tmp/loki/scratch
      alertmanager_url: https://alertmanager.xx
      external_url: https://alertmanager.xx

Expected behavior
Only specified streams should be deleted, otherwise the global policy is applied (8760h)

Environment:

Screenshots, Promtail config, or terminal output

@stale
Copy link

stale bot commented Jul 30, 2021

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale A stale issue or PR that will automatically be closed. label Jul 30, 2021
@cyriltovena
Copy link
Contributor

Hello !

That sound weird do you have more informations and example for stream that were deleted and shouldn't ?

@stale stale bot removed the stale A stale issue or PR that will automatically be closed. label Aug 4, 2021
@stale
Copy link

stale bot commented Sep 6, 2021

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.
We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.
Stalebots are also emotionless and cruel and can close issues which are still very relevant.
If this issue is important to you, please add a comment to reopen it.
More importantly, please add a thumbs-up to the original issue entry.
We regularly sort for closed issues which have a stale label sorted by thumbs up.
We may also:

  • Mark issues as revivable if we think it's a valid issue but isn't something we are likely
    to prioritize in the future (the issue will still remain closed).
  • Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

@stale stale bot added the stale A stale issue or PR that will automatically be closed. label Sep 6, 2021
@stale stale bot closed this as completed Sep 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/compactor feature/retention needs triage Issue requires investigation stale A stale issue or PR that will automatically be closed.
Projects
None yet
Development

No branches or pull requests

3 participants