Allocate memtable from the Arena

[Guyme]

Currently the speedb memtable does not allocate memory from the Arena.

[ayulas]

its not true! the memtable allocates the data from the arena!
the bucket is not allocates from the arena but i dont think it should

[udi-speedb]

@ofriedma - could you please give actual stats about the total memory allocations of our memtable, and show how much if from the arena and how much is not?

[ofriedma]

LOG-hashspdb-memory.txt
LOG-skiplist-memory.txt

At least from this log, there is a gap between the skiplist and our spdb hash, some memory is not transparent through the arena stats or the BlockFetcher stats in the hash spdb.

[udi-speedb]

LOG-hashspdb-memory.txt LOG-skiplist-memory.txt

At least from this log, there is a gap between the skiplist and our spdb hash, some memory is not transparent through the arena stats or the BlockFetcher stats in the hash spdb.

@ofriedma - Thanks.
Could you please summarise the highlights of your findings here.
Thanks

[Guyme]

First fix the memtable memory consumption - #618
Then add to the Arena (this ticket)

[ofriedma]

Summary from the logs:

SkipList memory information:

2023/11/08-16:23:32.972665 33053 [/db_impl/db_impl.cc:1138] --- Malloc STATS count=11986 memory=3292MB total=28017343
2023/11/08-16:23:32.972668 33053 [/db_impl/db_impl.cc:1141] Arena Stats: Arena total: 3402655080
2023/11/08-16:23:32.972789 33053 [/db_impl/db_impl.cc:1173] Total CacheAllocationUsage: 32654944

Hash spdb:

2023/11/08-16:35:58.474413 33460 [/db_impl/db_impl.cc:1138] --- Malloc STATS count=30610 memory=6040MB total=286460724
2023/11/08-16:35:58.474419 33460 [/db_impl/db_impl.cc:1141] Arena Stats: Arena total: 3592552848
2023/11/08-16:35:58.474950 33460 [/db_impl/db_impl.cc:1173] Total CacheAllocationUsage: 292628424

We can see there are differences between the allocations, higher memory consumption on hash spdb, that is not allocated through the arena.