Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Append to small bz1 blocks #3735

Merged
merged 1 commit into from
Aug 20, 2015

Conversation

benbjohnson
Copy link
Contributor

Overview

This pull request changes the bz1 append to check for a small ending block first. If the block is below the threshold for block size then it is rewritten with the new data points instead of having a new block written.

/cc @pauldix

@otoolep
Copy link
Contributor

otoolep commented Aug 19, 2015

@benbjohnson -- rebase your changes to pick up the test I have skipped.

@benbjohnson benbjohnson force-pushed the append-threshold branch 2 times, most recently from 5313724 to 06eb4b9 Compare August 19, 2015 18:59
@benbjohnson
Copy link
Contributor Author

@otoolep I rebased and updated the CHANGELOG.

@otoolep
Copy link
Contributor

otoolep commented Aug 19, 2015

Is this an optimization? It that its purpose?

@pauldix
Copy link
Member

pauldix commented Aug 19, 2015

@otoolep flushes can happen from the WAL due to memory pressure. The goal of this is to ensure that compressed blocks are big enough to actually get value from the compression. If you compress a single data point, your ratio won't be nearly as good as when you compress 1,000.

Also, lumping these together lessons the number of unique keys in the DB.

@otoolep
Copy link
Contributor

otoolep commented Aug 19, 2015

I see, a fix, makes sense @pauldix

@pauldix
Copy link
Member

pauldix commented Aug 19, 2015

@benbjohnson does the quick test cover all three scenarios? If so, +1

@benbjohnson
Copy link
Contributor Author

@pauldix It should but I'll add a more explicit append test for it.

@pauldix
Copy link
Member

pauldix commented Aug 19, 2015

+1 once that test is in. Also good to get this cherry picked into the 0.9.3 branch.

@benbjohnson benbjohnson force-pushed the append-threshold branch 2 times, most recently from 207b058 to f7ae6f8 Compare August 20, 2015 16:05
This commit changes the bz1 append to check for a small
ending block first. If the block is below the threshold
for block size then it is rewritten with the new data
points instead of having a new block written.
benbjohnson added a commit that referenced this pull request Aug 20, 2015
@benbjohnson benbjohnson merged commit 8f12cef into influxdata:master Aug 20, 2015
@benbjohnson benbjohnson deleted the append-threshold branch August 20, 2015 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants