Import shorten memtable switch latency

[Yuval-Ariel]

improve memtable switch latency with memtables that require large upfront allocation, such as as the hash-table-based memtable

Expected behavior

Actual behavior

Steps to reproduce the behavior

[erez-speedb]

Passed performance tests
Short
https://admin.speedb.io/performance?items=h0ENNbW9jKg6M5qMmbw5&items=YpWFlD2Gmno1wNZFGyaN&colors=%23F06292&colors=%2300796b
Small obj
https://admin.speedb.io/performance?items=wLA1gonLhMj7cXeyysrf&items=sgDc2TV9fp99lLZ4UCRE&colors=%23F06292&colors=%2300796b
Large obj
https://admin.speedb.io/performance?items=r5wzEWJxf8pjRh3wPOdS&items=8XHUSwkEKQRY3slFXvWo&colors=%23F06292&colors=%2300796b

The is some improvement in reads, no degradation in other tests and no additional memory consumption.

[assaf-speedb]

@Yuval-Ariel - 2 unit tests are getting constantly stuck:
db_write_buffer_manager_test-DBWriteBufferManagerTest-DBWriteBufferManagerTest.MixedSlowDownOptionsMultipleDB-0:
db_write_buffer_manager_test-DBWriteBufferManagerTest-DBWriteBufferManagerTest.MixedSlowDownOptionsMultipleDB-1:
stress and fuzz passed.

[Yuval-Ariel]

@Yuval-Ariel - 2 unit tests are getting constantly stuck: db_write_buffer_manager_test-DBWriteBufferManagerTest-DBWriteBufferManagerTest.MixedSlowDownOptionsMultipleDB-0: db_write_buffer_manager_test-DBWriteBufferManagerTest-DBWriteBufferManagerTest.MixedSlowDownOptionsMultipleDB-1: stress and fuzz passed.

@ayulas , can you look at the tests?

[ayulas]

@assaf-speedb please retest

[ayulas]

please verify the hash is 7115af7

[assaf-speedb]

Unit tests, stress, and fuzz - all passed

[isaac-io]

I'm seeing the following failures on main after the merge of #25:

db_memtable_test

Note: Google Test filter = DBMemTableTest.InsertWithHint
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DBMemTableTest
[ RUN      ] DBMemTableTest.InsertWithHint
db/db_memtable_test.cc:303: Failure
Expected: (hint_foo) != (hint_bar), actual: NULL vs NULL
terminate called after throwing an instance of 'testing::internal::GoogleTestFailureException'
  what():  db/db_memtable_test.cc:303: Failure
Expected: (hint_foo) != (hint_bar), actual: NULL vs NULL
Received signal 6 (Aborted)
#0   /usr/lib/libc.so.6(+0x8e36c) [0x7f1bacc8e36c] ??	??:0	
#1   /usr/lib/libc.so.6(raise+0x18) [0x7f1bacc3e838] ??	??:0	
#2   /usr/lib/libc.so.6(abort+0xcf) [0x7f1bacc28535] ??	??:0	
#3   /usr/lib/libstdc++.so.6(+0x99833) [0x7f1bad099833] ??	??:0	
#4   /usr/lib/libstdc++.so.6(+0xa5bfc) [0x7f1bad0a5bfc] ??	??:0	
#5   /usr/lib/libstdc++.so.6(+0xa5c69) [0x7f1bad0a5c69] ??	??:0	
#6   /usr/lib/libstdc++.so.6(__cxa_rethrow+0x4f) [0x7f1bad0a5f1f] ??	??:0	
#7   ./db_memtable_test(+0x138101) [0x563328456101] ??	??:0	
#8   ./db_memtable_test(+0x12ae94) [0x563328448e94] ??	??:0	
#9   ./db_memtable_test(+0x93ded) [0x5633283b1ded] ??	??:0	
#10  /usr/lib/libc.so.6(+0x29290) [0x7f1bacc29290] ??	??:0	
#11  /usr/lib/libc.so.6(__libc_start_main+0x8a) [0x7f1bacc2934a] ??	??:0	
#12  ./db_memtable_test(+0xbd895) [0x5633283db895] ??	??:0	
t/run-db_memtable_test-DBMemTableTest.InsertWithHint: line 4: 4040866 Aborted                 (core dumped) TEST_TMPDIR=$d ./db_memtable_test --gtest_filter=DBMemTableTest.InsertWithHint

db_test

Note: Google Test filter = DBTest.DynamicMemtableOptions
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DBTest
[ RUN      ] DBTest.DynamicMemtableOptions
db/db_test.cc:4364: Failure
Expected equality of these values:
  NumTableFilesAtLevel(0)
    Which is: 2
  1
terminate called after throwing an instance of 'testing::internal::GoogleTestFailureException'
  what():  db/db_test.cc:4364: Failure
Expected equality of these values:
  NumTableFilesAtLevel(0)
    Which is: 2
  1
Received signal 6 (Aborted)
#0   /usr/lib/libc.so.6(+0x8e36c) [0x7fddc4e8e36c] ??	??:0	
#1   /usr/lib/libc.so.6(raise+0x18) [0x7fddc4e3e838] ??	??:0	
#2   /usr/lib/libc.so.6(abort+0xcf) [0x7fddc4e28535] ??	??:0	
#3   /usr/lib/libstdc++.so.6(+0x99833) [0x7fddc5299833] ??	??:0	
#4   /usr/lib/libstdc++.so.6(+0xa5bfc) [0x7fddc52a5bfc] ??	??:0	
#5   /usr/lib/libstdc++.so.6(+0xa5c69) [0x7fddc52a5c69] ??	??:0	
#6   /usr/lib/libstdc++.so.6(__cxa_rethrow+0x4f) [0x7fddc52a5f1f] ??	??:0	
#7   ./db_test(+0x231901) [0x5621f8a62901] ??	??:0	
#8   ./db_test(+0x224b34) [0x5621f8a55b34] ??	??:0	
#9   ./db_test(+0xbe7a8) [0x5621f88ef7a8] ??	??:0	
#10  /usr/lib/libc.so.6(+0x29290) [0x7fddc4e29290] ??	??:0	
#11  /usr/lib/libc.so.6(__libc_start_main+0x8a) [0x7fddc4e2934a] ??	??:0	
#12  ./db_test(+0xed135) [0x5621f891e135] ??	??:0	
t/run-db_test-DBTest.DynamicMemtableOptions: line 4: 4092448 Aborted                 (core dumped) TEST_TMPDIR=$d ./db_test --gtest_filter=DBTest.DynamicMemtableOptions

options_util_test

Note: Google Test filter = OptionsUtilTest.LoadLatestOptions
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from OptionsUtilTest
[ RUN      ] OptionsUtilTest.LoadLatestOptions
options_util_test: db/memtable.cc:117: rocksdb::MemTable::MemTable(const rocksdb::InternalKeyComparator&, const rocksdb::ImmutableOptions&, const rocksdb::MutableCFOptions&, rocksdb::WriteBufferManager*, rocksdb::SequenceNumber, uint32_t, bool): Assertion `!ShouldScheduleFlush()' failed.
t/run-options_util_test-OptionsUtilTest.LoadLatestOptions: line 4: 253382 Aborted                 (core dumped) TEST_TMPDIR=$d ./options_util_test --gtest_filter=OptionsUtilTest.LoadLatestOptions

That last one seems to only fail when running through make check, but not when run directly. However, with make check it consistently fails. I think we need to revert.

[ayulas]

all test failure are a test issues
db_memtable_test:

in the MockMemTable the last_hint_in last_hint_out are not initialized so it may have garbage value
also if a switch memtable occur the check on theses value is irrelevant....

db_test:
it is not aligned to the fact we have an active memtable and a pending memtable...
so the check of the L0 files (menaing the memtable that have done flush) depending the increase/decrease write buffer size is incorrect cause the pending memtable is created with the previous write buffer size ...

options_util_test:
i ran the make check and didnt get this failure . is it a local run on your laptop?

[Yuval-Ariel]

QA passed on 62b3f64

[erez-speedb]

Past performance tests on hash 7e5e7bc

[erez-speedb]

Also passed QA

[Guyme]

Switch functionality was added to ticket #98 PR #188 7e5e7bc