Hybrid logstore

[dgtony]

Currently logstore requests are skewed towards the reads, while some retrievals (notably GetThread) are pretty expensive, requiring multiple subrequests of logstore components that goes through the series of mutexes preserving data consistency.
So we implemented a new hybrid model of logstore, which is both persistent and enables fast reads. Basically it's just a write-through in-memory cache backed by a datastore for persistency. We combined two existing implementations (lstoreds + lstoremem), such that all reads goes into in-memory store only, and writes are performed in both.
Resulting hybrid model makes request processing order of magnitude faster with a tradeoff of slower writes and increased memory consumption as we're keeping all the data in memory. Given that such a setting is not universal but would be useful for higher-rank nodes, it can be enabled with an option and lstoreds remains the default one.
As a useful side effect, now we also get export/import capabilities for all the different logstore implementations. It makes implementation of components and data exchange easier, enables dumping data for analysis etc.

[sanderpick]

Hey @dgtony, looking good! I started looking though it. Will take another look this evening. Happy to see the importer / exporter functionality. That was actually on my to-do list 🎉

[sanderpick]

Good idea! I didn't dive super deep into the dump/restore logic for all the books, but will be playing around with that functionality soon and will look closer.

Any ideas about the test failures?

[sanderpick]

Nice, clever.

[sanderpick]

How about LogstoreType?

[dgtony]

renamed

[dgtony]

Regarding test failures. This one and some other random failures start periodically emerge on the master. Looks like a timing issue, e.g. introducing the 20 milliseconds sleep before checking test results (db/collection_test.go:931) seems to fix it.