Skip to content

Commit

Permalink
use DisableIndexDeduplication which disables deduplication of just th…
Browse files Browse the repository at this point in the history
…e index and still dedupes chunks
  • Loading branch information
sandeepsukhani committed Jul 24, 2020
1 parent 62e7299 commit 1ed418b
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
4 changes: 2 additions & 2 deletions docs/sources/operations/storage/boltdb-shipper.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ The problem with write deduplication when using `boltdb-shipper` though is inges
The problem due to that is if an ingester which first wrote the chunks and index goes down and all the other ingesters which were part of replication scheme skipped writing those chunks and index due to deduplication, we would end up missing those logs from query responses since only the ingester which had the index went down.
This problem would be faced even during rollouts which is quite common.

To avoid this, Loki disables WriteDedupe and uses Chunks cache only for improving read performance when the replication factor is greater than 1 and `boltdb-shipper` is an active or upcoming index type.
While Chunks cache would still be useful to have when using `boltdb-shipper` for read performance, please avoid configuring WriteDedupe cache since it would not be used anyways.
To avoid this, Loki disables deduplication of index when the replication factor is greater than 1 and `boltdb-shipper` is an active or upcoming index type.
While using `boltdb-shipper` please avoid configuring WriteDedupe cache since it is used purely for the index deduplication, so it would not be used anyways.


2 changes: 1 addition & 1 deletion pkg/loki/modules.go
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ func (t *Loki) initStore() (_ services.Service, err error) {
// If RF > 1 and current or upcoming index type is boltdb-shipper then disable both chunks dedupe and write dedupe cache.
// This is to ensure that index entries are replicated to all the boltdb files in ingesters flushing replicated data.
if t.cfg.Ingester.LifecyclerConfig.RingConfig.ReplicationFactor > 1 && usingBoltdbShipper(t.cfg.SchemaConfig) {
t.cfg.ChunkStoreConfig.DisableChunksDeduplication = true
t.cfg.ChunkStoreConfig.DisableIndexDeduplication = true
t.cfg.ChunkStoreConfig.WriteDedupeCacheConfig = cache.Config{}
}

Expand Down

0 comments on commit 1ed418b

Please sign in to comment.