rocksdb high cpu usage due to LRU cache contention?

[zaidoon1]

We can see a strong correlation between increasing the number of block cache HITs and the amount of cpu used by rocksdb:

Is this expected behaviour? And is there a fix for this? Is switching to hyper clock cache the only way to truly fix this?

[mdcallag]

The big win comes from using hyper clock.

Smaller improvements might be possible from using more shards in the LRU block cache assuming that your problem is a hot shard rather than a hot block.

[zaidoon1]

got it, that's good to know! I've started rolling out the change to use hyper clock cache. I'll keep this issue open for now and will post a before/after

[zaidoon1]

Before (LRU cache):

After (Hyper clock cache):

Very happy with the results here. Thanks!