The TSM storage engine #4308
Merged
The TSM storage engine #4308
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Also fixed shard to work again with b1 and bz1 engines.
Also ensure that queries don't try to use files that have been deleted.
Prevents index out of bounds panic
Prevents a panics when response size is less than 100. Also allows data to be posted when it is less than the batch size.
Time compression uses an adaptive approach using delta-encoding, frame-of-reference, run length encoding as well as compressed integer encoding. Float compression uses an implementation of the Gorilla paper encoding for timestamps based on XOR deltas and leading and trailing null suppression.
This is using zig zag encoding to convert int64 to uint64s and then using simple8b to compress them, falling back to uncompressed if the value exceeds 1 << 60. A patched encoding scheme would likely be better in general but this provides decent compression for integers that are not at the ends of the int64 range.
Shard path can be a directory.
Not implemented for tsm1 engine
Will make it less error-prone to add new encodings int the future since each encoder has it's set of constants. There are some placeholder contants for uncompressed encodings which are not in all encoder currently.
Avoid panicing in lower level code and allow the engine to decide what it should do.
Should never get a block size 9 bytes since Encode always returns the min timestampe and a 1 byte header. If we get this, the engine is confused.
… pd-storage-engine
Closing the store did not properly return an error for in-flight writes because the closing channel was set to nil when closed. A nil channel is not selectable so writes continue on past the guard checks and trigger panics.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Internal people have seen the docs, external will see it soon. I haven't had a chance to refactor any of this yet, but let me have it. I'll be pushing up some refactoring changes over the weekend I think.
Will fix the merge conflicts with the stress tool in a bit, but I wanted to get people commenting on the code so I can get some feedback and fixes in.