Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FIFO Cache validity behavior and lack of auto expiration #4921

Closed
1 of 2 tasks
slim-bean opened this issue Dec 13, 2021 · 0 comments · Fixed by #5148
Closed
1 of 2 tasks

FIFO Cache validity behavior and lack of auto expiration #4921

slim-bean opened this issue Dec 13, 2021 · 0 comments · Fixed by #5148

Comments

@slim-bean
Copy link
Collaborator

slim-bean commented Dec 13, 2021 •
edited
Loading

As part of the 2.4 release we enabled the in memory FIFO Cache by default and set a max size (1GB) and max validity (1h) in the default configs.

I realized observing some running Loki instances that the memory was higher than expected and not reduced over time.

The querier_cache_memory_bytes showed that the chunks cache was full at 1GB and never reduces.

After digging, the validity config does not behave how I expected, it's only read when a Get is called on the cache to determine if the value should be returned to the caller

However:

  • It does not prune invalid entries even on a GET call
  • It does not prune old entries on any kind of timer.

I think we should probably do two things
@slim-bean slim-bean mentioned this issue Dec 13, 2021
Closed
@slim-bean slim-bean mentioned this issue Jan 6, 2022
Closed
chaudum added a commit that referenced this issue Jan 14, 2022
@chaudum
Add periodic task to prune old FIFO cache elements
0b6fa94 
As described in #4921 this PR adds a periodic task that prunes expired
items from the FIFO cache to free up memory.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@chaudum chaudum mentioned this issue Jan 14, 2022
Merged
3 tasks
chaudum added a commit that referenced this issue Jan 17, 2022
@chaudum
Add periodic task to prune old FIFO cache elements
a159560 
As described in #4921 this PR adds a periodic task that prunes expired
items from the FIFO cache to free up memory.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Jan 18, 2022
@chaudum
Add periodic task to prune old FIFO cache elements
f13e556 
As described in #4921 this PR adds a periodic task that prunes expired
items from the FIFO cache to free up memory.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Jan 24, 2022
@chaudum
Add periodic task to prune old FIFO cache elements
7f00665 
As described in #4921 this PR adds a periodic task that prunes expired
items from the FIFO cache to free up memory.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Jan 29, 2022
@chaudum
Add periodic task to prune old FIFO cache elements
ac3537d 
As described in #4921 this PR adds a periodic task that prunes expired
items from the FIFO cache to free up memory.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
owen-d pushed a commit that referenced this issue Jan 31, 2022
@chaudum @KMiller-Grafana
Auto-expire old items from FIFO cache (#5148)
57b384d 
* Remove validity from FIFO cache

We can remove the validity setting of 1h in 2.4 for the
chunks cache because it doesn't reduce memory usage and instead leads to
valid chunks being ignore when querying the cache. Chunks are immutable
so it doesn't make sense to not return them if they are present in the
cache.

Closes #4922

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

* Add periodic task to prune old FIFO cache elements

As described in #4921 this PR adds a periodic task that prunes expired
items from the FIFO cache to free up memory.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

* Apply suggestions from code review

Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>

Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Auto-expire old items from FIFO cache grafana/loki
1 participant
@slim-bean