Internal Lock Granularities vs Optimal Application Thread Count

[victorstewart]

wondering if you've done any experiments around the maximum number of threads that can be used before contention makes sharding into another process advisable? the basis of my question assumes there might be some global write locks, but maybe only per new key value pair?

[beckerg]

Hi Victor,

We have put quite a bit of effort into identifying and reducing contention in HSE, employing different techniques depending upon the situation, but I don't have any data to share regarding scalability (although I can say that HSE does scale extremely well, in general). Is there a particular use-case I can address? In general, the number and size of keys will directly affect throughput and latency of both get and put operations (e.g., a KVDB that fits entirely within RAM will outperform a KVDB that does not fit into available RAM).

For the most part, contention in the get/put paths is diminished by using per-CPU structures and algorithms to accomodate machines with high CPU counts. Probably the largest point of contention in HSE is with transaction commit, which must serialize commits across the entire KVDB (i.e., amongst all KVS'). At the front end of a transaction is the need to acquire a stable view and then maintain the view horizon after commit, which also must be done across the entire KVDB. All this code is highly optimized for throughput, but are choke points none-the-less, and fertile grounds for improvements.

Regarding sharding, it's difficult to answer. I would start with a single KVDB to try and identify whether you are CPU, memory, and/or IO bound, and then go from there. I would definitely be interested in knowing details about any performance issues that cause you grief, so please let us know!

At any rate, I can give you more detail about the locking in various parts of the HSE machinery, but it would be easiest to address only the parts that are affecting your use cases.

Greg