Enable the jemalloc option --disable-cache-oblivious

[aleksraiden]

Inspired an Redis release notice, I add a two options for optimize jmalloc usages:

• disable-tcache - Disable thread-specific caches for small objects. Objects are cached and released in bulk, thus reducing the total number of mutex operations.
• disable-cache-oblivious - Disable cache-oblivious large allocation alignment for large allocation requests with no alignment constraints. If this feature is disabled, all large allocations are page-aligned as an implementation artifact, which can severely harm CPU cache utilization. For more info, please see a conversations in Redis - Set Jemalloc --disable-cache-oblivious to reduce memory overhead redis/redis#12315 where this options are available started at 7.3-rc2.

[mapleFU]

I think we should testing rather than copying the config directly from Redis. Redis store all parts in memory, and has too many small objects, like sds, skiplist node and so on. So small object would be a severe problem for it.
However, kvrocks stores these objects on a log-structure merge tree, and data is organized as sst/blocks. I guess there would be something different.

[aleksraiden]

In my quick tests, these changes brought a slight improvement in terms of better freeing up memory when kvroks is idle.

[git-hulk]

@aleksraiden Could you paste your test result here?

[aleksraiden]

I haven't a clear env, but overall, if I run ./redis-benchmark -q -n 1000000 -P 10 -p 6666 my instance of kvrocks use around a 505 Mb RAM after a hour of idle (after a test are stopped). And, jmalloc-fix version use 432 Мб in this case too.

[git-hulk]

@aleksraiden Thanks for your information. I feel good after looking through jemalloc/jemalloc#1098. Another question is why disable the tcache as well, did I miss anything?

[aleksraiden]

@aleksraiden Thanks for your information. I feel good after looking through jemalloc/jemalloc#1098. Another question is why disable the tcache as well, did I miss anything?

As I think - because we don't need an a global cache for thread-specific objects, so if a small object usages only in one thread - this thread can feel free to manage this and release memory without use mutex for thread-safe operations.

[git-hulk]

@aleksraiden Thanks for your information. I feel good after looking through jemalloc/jemalloc#1098. Another question is why disable the tcache as well, did I miss anything?

As I think - because we don't need an a global cache for thread-specific objects, so if a small object usages only in one thread - this thread can feel free to manage this and release memory without use mutex for thread-safe operations.

I prefer to keep it as it is unless we can prove that can bring benefits.

[aleksraiden]

@aleksraiden Thanks for your information. I feel good after looking through jemalloc/jemalloc#1098. Another question is why disable the tcache as well, did I miss anything?

As I think - because we don't need an a global cache for thread-specific objects, so if a small object usages only in one thread - this thread can feel free to manage this and release memory without use mutex for thread-safe operations.

I prefer to keep it as it is unless we can prove that can bring benefits.

Redis use this options too, but if reviewers are OK, I can update PR and remove option about tcache

[git-hulk]

Redis use this options too

I didn't notice this option, would you mind giving me a link?

but if reviewers are OK, I can update PR and remove option about tcache

It's good for me to add if we can know/prove it did bring benefits.

[aleksraiden]

Redis use this options too

I didn't notice, would you mind giving me a link?

but if reviewers are OK, I can update PR and remove option about tcache

It's good for me to add if we can know/prove it did bring benefits.

Oops, I'am SORRY, my mistake, just think about one options, but write for other, Yes, you are absolutely correct, Redis didn't use this options about tcache disabling.

[mapleFU]

1. disable-tcache: I guess this might not benefits because at most of time, thread cache is an optimization
2. as for disable-cache-oblivious, redis says it would only benefits 16kb allocations, however, I guess kvrocks could have many over 16kb allocation because of rocksdb.

Maybe we can take a look at TiKV's arguments: https://github.com/tikv/jemallocator/blob/main/jemalloc-ctl/src/opt.rs . It also uses RocksDB as storage engine.

[git-hulk]

Oops, I'am SORRY, my mistake, just think about one options, but write for other, Yes, you are absolutely correct, Redis didn't use this options about tcache disabling.

@aleksraiden Thanks for your investigation.

Maybe we can take a look at TiKV's arguments: https://github.com/tikv/jemallocator/blob/main/jemalloc-ctl/src/opt.rs . It also uses RocksDB as storage engine.

@mapleFU That's a good input.

[mapleFU]

Seems that by default, if we disable cache-oblivious, we will

1. Not align the large chunk, which make 16kb page just uses 16kb of memory, rather than about 20kb, which brings more usable memory to fill the "block cache" required by system
2. May harms some cache-obilivious. But seems that we don't have that

I guess maybe we can allow enable that for some read-heavy systems, that tent to fill the memory with block cache, so this is useful. Rocksdb might not have cache-oblivious algorithm, so it might not harms too much?

But I guess for most of workloads, they might have downgrade or not, because we have this https://github.com/facebook/rocksdb/blob/854eb76a8c8533779da7eaf21bef6266d9cca6aa/memory/arena.h#L32-L33 block size, it could become 16KB.

Maybe we can support a flag, and let different kind of user to specify their usage.

Another view is that Redis is a single-thread application, so most of time, it doesn't matters. If we care about the multi-core performance, it might harm the application.

[mapleFU]

Seems that CI compile failed :-)

[aleksraiden]

Updated. So, now we can set options in build, e.g. -D=DISABLE_CACHE_OBLIVIOUS=0|1 where 0 are default and only set to 1 add this options into jemalloc compile process.

[aleksraiden]

Thanks @torwig for help with cmake

[git-hulk]

Thanks all, merging...