Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement data retention in storage layer #168

Closed
2 of 6 tasks
petethepig opened this issue Apr 28, 2021 · 0 comments · Fixed by #182
Closed
2 of 6 tasks

Implement data retention in storage layer #168

petethepig opened this issue Apr 28, 2021 · 0 comments · Fixed by #182
Labels
backend Mostly go code storage Low level storage matters

Comments

@petethepig
Copy link
Member

petethepig commented Apr 28, 2021

Feature Description

Sometimes it doesn't make sense to store data indefinitely so we need to implement a way to do data retention.

Internally Pyroscope uses segment trees to store the data.
See our Storage Design Doc for more context.

Page2 3

We need to add a new retention threshold parameter that would control how much data (in days) can be stored in pyroscope. Then we would run some sort of a Cleanup method every once in a while to delete old data:

Page3

TODO

  • Implement a method that would delete data before a certain time on segment tree and return a list of trees that need also need to be deleted.
  • cover it with specs
  • add a retention option to config
  • in storage package, put conditions in Put method that would prevent data ingestions past the retention threshold
  • in storage package add a method that would delete segment tree nodes as well as relevant trees
  • create a separate goroutine that would call this cleanup method every X minutes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Mostly go code storage Low level storage matters
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant