Faster gets and query/range retrieval

[kriszyp]

This release includes a significant overhaul of how get and query/range retrievals are performed to reduce the amount of native calls and the number of JS objects that have to be transmitted across the native calls. This also facilitates and accompanies adding support for V8's new fast-api-calls feature, which also improves performance. This can yield over a 50% performance improvement in get operations, and over twice the speed for iterating through ranges, with small payloads where deserialization is not the dominant cost. There are also very large improvements in the performance of count operations and offset handling as well.

Also related to these changes is switching (back) to serializing and deserializing keys in JS (instead of C++), which reduces the number of JS value/objects cross the native call barrier. This also opens the door for more customized JS key serialization strategies (negation, custom UUID handlers, and potentially more performant little endian format).

---

This discussion was created from the release Faster gets and query/range retrieval.

[vladar]

This is awesome 👍

I did some benchmarks and we see 20% speed-up for real-word queries with lmdb-store@1.6.0-beta.1. And especially significant improvement in counts (not using #66 yet - just counting by key range):

(higher is better)

The benchmark runs something like SELECT Z FROM X WHERE Y = "FOO" LIMIT 100 against an index. The filter statement captures about 1/4 of all "rows".

3 columns on the very left are from our pure in-memory store (basically JS Maps and Arrays). Just as a baseline. Middle 3 is lmdb-store@1.5.5, last 3 is lmdb-store@1.6.0-next.1 (first column - 10k rows, 2nd - 50k rows, 3rd - 100k rows)

Various series within column show variations of the same query:

• COUNT=1 adds a count of total rows to the final result (e.g. returns 2500 for 10000 total nodes in addition to the main query)
• SORT=1 adds sort by some random field (with LMDB sort field is in the index, with in-memory we re-sort every time)
• TEXT=1 adds 4kb of text to each "row" which is included in the result

Also the result of each query is written to disk, so store and index is not the only factor in play, but we needed to see the impact close to the real world.

Pretty impressive results in comparison to in-memory store. Esp. happy to see counts improved so much. Looking forward to 1.6.0 release and thank you for such a fantastic package!

[kriszyp]

Wow, it is really cool to see some great independent benchmarking! Thank you for sharing, it is great to get to hear how this is being used and that the performance gains are adding up!