Improve Compaction Performance

[e-dard]

This PR adds new batch-oriented tsm1.Encoders that aim to improve the encoding performance of the various TSM blocks (Float, Integer, Unsigned, Time, String and Boolean) with a more efficient implementation of the various encoding schemes.

No encoding/compression behaviour had been altered, and these encoders are compatible with the existing tsm1 format.

By improving the performance of the encoders, the aim is that compactions themseleves will see improved performance.

This PR concerns itself with the following changes:

• ac90e5f: a batch oriented float encoder (FloatArrayEncodeAll) using Gorilla compression, but avoiding the function call and poor cache effects of using the existing bitstream package. Instead, we take all the float values to encode, and encode them all at once. Benchmarks show an improvement of around 5x - 16x in encoding performance for a 1,000 value block, when compared to the existing encoder implementation. (See a2724e2 for the updated benchmarks)

• 5a5ffe6: batch oriented integer/unsigned integer encoders (IntegerArrayEncodeAll and UnsignedArrayEncodeAll) using either uncompressed, RLE or simple8b compression. Benchmarks show an improvement of around 1.5x - 1.7x in encoding performance for a 1,000 value block.

• 5decd99: a batch oriented timestamp encoder (TimeArrayEncodeAll) using either uncompressed, RLE or simple8b compression. Benchmarks show an improvement of around 1.5x - 2.5x in encoding performance for a 1,000 value block. These improvements carry to all blocks, since they all encode timestamps.

• dd13d79 : a batch oriented string encoder (StringArrayEncodeAll). Benchmarks show an improvement of around 13% – 66% in encoding performance for a 1,000 value block. The original encoder was already quite performant.

• 0a5eb7a: add block encoders for all types that encode the values and timestamps into the tsm block.

• 902ba41 add a new key iterator tsmBatchKeyIterator that uses the batch decode and encode functions to compact TSM blocks.

Not implemented

There are several things that have not been implemented in this PR, and could be considered future work:

• Pooling of the buffers used for the encoding. I put a lot of effort into reusing the provided buffer. However, it's a non trivial amount of work to create the memory pools necessary to reuse those buffers. We need to know for example when the buffer has been safely written to disk so we can mutate the buffer. This could be future work.
• Cache snapshots. The snapshots do not currently use the new encoders. They could however.
• Encoding in some offline tooling (though the offline compactor will use these changes).

TODO

I need to add some compaction benchmarks here....

[benbjohnson]

I can't say I fully grok the bit twiddling but the test suite looks solid. Just a few nits but otherwise lgtm.

[benbjohnson]

fir?

[benbjohnson]

Remove comment.

[benbjohnson]

Would using testing/quick make more sense so you could adjust the number of iterations and control the determinism of the PRNG?

[benbjohnson]

This else is not needed since you're returning in the previous block.

[benbjohnson]

You can unindent the code in the block by changing this to:

if len(v) != 0 {
	continue
}

[stuartcarnie]

Looking great Edd, nice work!

I pushed a couple more improvements to the integer and string encoders on Friday afternoon. Will be happy to contribute more tests or do whatever we need to verify stability of the array encoders and decoders. I'll follow up with a PR for the Float decoder.