Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimise LSM compaction for topic data #271

Open
purplefox opened this issue Oct 3, 2024 · 0 comments
Open

Optimise LSM compaction for topic data #271

purplefox opened this issue Oct 3, 2024 · 0 comments
Milestone

Comments

@purplefox
Copy link
Contributor

LSM can be configured to be more efficient for topic data (append only).

  • Make SSTable size larger in lower levels - e.g. double size of table (up to a limit) on successive levels. This will reduce number of PUTs to S3 in compaction, thus reducing costs. It also makes historic consumers require less GETs.
  • Consider making L1 max size very large. I.e. effectively have just two levels: L0 and L1 in the LSM. This will reduce number of S3 writes as compaction would typically only be done once. It can still be efficient as even if L1 gets very large, overlap of L0 to L1 can be small as data is append only.
@purplefox purplefox added this to the 1.0 milestone Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant