Logs disappear after the chunks were removed from memory

[putrasattvika]

Describe the bug
Logs that have been ingested disappear after the chunks were removed from the
ingesters' memory. Before the chunks are flushed and removed, all of the logs
are queryable via logcli and Grafana.

Reproduced with commit hash fa79c2469c59c90cfb8e0415b0998665d5db2601
in single process mode, but also happened in microservice mode with S3-DynamoDB
and Cassandra-Cassandra index-chunk storage.

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (fa79c24)
2. Started Promtail (fa79c24) to tail random2.txt
3. Query with logcli: {filename!=""}, there are 450 log lines as expected
4. Wait until the loki_ingester_memory_chunks metric dropped to 0
5. Query with logcli: {filename!=""}, there are less than 450 log lines. The remaining log lines are here.

Expected behavior
The number of logs returned after the chunks are removed from memory to be equal
with when the logs haven't been flushed yet.

Environment:

• Infrastructure:

  • Linux 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • go version go1.13.3 linux/amd64

• Deployment tool: single process mode

Screenshots, Promtail config, or terminal output

logcli outputs:

$ date; logcli --addr=http://localhost:3100 query '{filename!=""}' --limit 1000 | wc -l
Mon 21 Oct 18:10:59 WIB 2019
http://localhost:3100/loki/api/v1/query_range?direction=BACKWARD&end=1571656259418474786&limit=1000&query=%7Bfilename%21%3D%22%22%7D&start=1571652659418474786
Common labels: {filename="/home/isattvika/workspace/personal/2019-10-01_centralized-logging/random2.log", kube_container="my-container", kube_namespace="my-service", kube_replicaset="my-service-569d4f4f85"}
450

$ date; logcli --addr=http://localhost:3100 query '{filename!=""}' --limit 1000 | wc -l
Mon 21 Oct 18:13:22 WIB 2019
http://localhost:3100/loki/api/v1/query_range?direction=BACKWARD&end=1571656402404992865&limit=1000&query=%7Bfilename%21%3D%22%22%7D&start=1571652802404992865
Common labels: {filename="/home/isattvika/workspace/personal/2019-10-01_centralized-logging/random2.log", kube_container="my-container", kube_namespace="my-service", kube_replicaset="my-service-569d4f4f85"}
109

Loki configuration:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  max_transfer_retries: 1
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1m
  chunk_retain_period: 30s

schema_config:
  configs:
  - from: 2019-10-01
    store: boltdb
    object_store: filesystem
    schema: v10
    index:
      prefix: index_
      period: 168h

storage_config:
  boltdb:
    directory: /tmp/loki/boltdb/index

  filesystem:
    directory: /tmp/loki/boltdb/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0

table_manager:
  chunk_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  index_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  retention_deletes_enabled: false
  retention_period: 0

Promtail config:

server:
  http_listen_port: 10080
  grpc_listen_port: 19095

clients:
  - url: http://localhost:3100/loki/api/v1/push

positions:
  filename: /home/isattvika/programs/promtail/positions.yml

scrape_configs:
  - job_name: testing

    static_configs:
      - targets:
          - localhost

        labels:
          __path__: /home/isattvika/workspace/personal/2019-10-01_centralized-logging/random2.log

    pipeline_stages:
      - json:
          expressions:
            message:
            timestamp:
            labels:

      - json:
          expressions:
            kube_container: container
            kube_pod: pod
            kube_replicaset: replicaset
            kube_namespace: namespace
            kube_node: node
          source: labels

      - labels:
          kube_container:
          kube_pod:
          kube_replicaset:
          kube_namespace:
          kube_node:

      - timestamp:
          source: timestamp
          format: 2006-01-02T15:04:05.000000Z

      - output:
          source: message

[k2glyph]

Any update. I am also having same issue.

[megastallman]

Please reopen. Affects versions 1.6.0 and 1.6.1 at least. I can read and write logs to Loki, but within a minute. Then they're lost.
Here is my /etc/loki/loki.yaml

auth_enabled: false
chunk_store_config:
  max_look_back_period: 0s
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  max_transfer_retries: 0
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
schema_config:
  configs:
  - from: "2018-04-15"
    index:
      period: 168h
      prefix: index_
    object_store: filesystem
    schema: v9
    store: boltdb
server:
  http_listen_port: 3100
storage_config:
  boltdb:
    directory: /data/loki/index
  filesystem:
    directory: /data/loki/chunks
table_manager:
  retention_deletes_enabled: true
  retention_period: 672h

Can also omit the table_manages section.

[megastallman]

#1772 - this was my problem actually.
So, moved to /loki/api/v1/query_range and everything works.