Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(TSDB): allow tsdb index creation in memory only #14732

Merged
merged 11 commits into from
Nov 4, 2024

Conversation

owen-d
Copy link
Member

@owen-d owen-d commented Nov 1, 2024

In preparation for creating stateless ingesters, this PR adds support for TSDBs to be created by using memory as the backing temporary store rather than disk. This paves the way towards an easier operating model which doesn't require disks.

Note: this adds support but doesn't use it in any existing code paths. I've essentially refactored the TSDB creator to use an interface which can be satisfied by disk or memory. The old code paths will continue to use disks for now to minimize changes.

# index files built pre and post change are byte equivalent
$ cmp 1730487398326535000-compactor-1000-3000-bd34932f.tsdb 1730487451442167000-compactor-1000-3000-bd34932f.tsdb

ReaderFrom & Writer impl
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
@owen-d owen-d requested a review from a team as a code owner November 1, 2024 18:17
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
// serialization format.
type Writer struct {
type Creator struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Writer was still fine.

Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@owen-d owen-d merged commit 831c0d5 into grafana:main Nov 4, 2024
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants