Server asserts on startup with rocksdb-persistent-cache-size-mb

[george-lorch]

Nothing fancy to reproduce.
my.cnf :
[mysqld] basedir=/ssd/percona/myrocks/facebook-mysql-install-5.6 datadir=/ssd/percona/myrocks/facebook-mysql-install-5.6/data tmpdir=/ssd/percona/myrocks/facebook-mysql-install-5.6/tmp port=10000 socket=/ssd/percona/myrocks/facebook-mysql-install-5.6/var/mysql.sock pid-file=/ssd/percona/myrocks/facebook-mysql-install-5.6/var/mysql.pid console server-id=1 max_connections=1000 rocksdb default-storage-engine=rocksdb skip-innodb default-tmp-storage-engine=MyISAM rocksdb-persistent-cache-size-mb=30

Backtrace :
#0 0x00007ffff5f53c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff5f57028 in __GI_abort () at abort.c:89
#2 0x00007ffff5f4cbf6 in __assert_fail_base (fmt=0x7ffff60a1018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x1de1966 "status.ok()",
file=file@entry=0x1de1900 "/ssd/percona/myrocks/facebook-mysql-5.6/rocksdb/utilities/persistent_cache/block_cache_tier.cc", line=line@entry=33,
function=function@entry=0x1de31a0 rocksdb::BlockCacheTier::Open()::__PRETTY_FUNCTION__ "virtual rocksdb::Status rocksdb::BlockCacheTier::Open()") at assert.c:92
#3 0x00007ffff5f4cca2 in __GI___assert_fail (assertion=0x1de1966 "status.ok()", file=0x1de1900 "/ssd/percona/myrocks/facebook-mysql-5.6/rocksdb/utilities/persistent_cache/block_cache_tier.cc", line=33,
function=0x1de31a0 rocksdb::BlockCacheTier::Open()::__PRETTY_FUNCTION__ "virtual rocksdb::Status rocksdb::BlockCacheTier::Open()") at assert.c:101
#4 0x000000000160763e in rocksdb::BlockCacheTier::Open (this=0x7fffd27cd018) at /ssd/percona/myrocks/facebook-mysql-5.6/rocksdb/utilities/persistent_cache/block_cache_tier.cc:33
#5 0x0000000001609dd6 in rocksdb::NewPersistentCache (env=0x27d0860 rocksdb::Env::Default()::default_env, path="", size=31457280, log=std::shared_ptr (count 6, weak 0) 0x7ffff5629a78, optimized_for_nvm=true,
cache=0x7fffffffcac0) at /ssd/percona/myrocks/facebook-mysql-5.6/rocksdb/utilities/persistent_cache/block_cache_tier.cc:413
#6 0x00000000012b9722 in myrocks::rocksdb_init_func (p=0x7ffff54bac00) at /ssd/percona/myrocks/facebook-mysql-5.6/storage/rocksdb/ha_rocksdb.cc:3977
#7 0x0000000000d56a24 in ha_initialize_handlerton (plugin=0x7fffd23fbe30) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/handler.cc:655
#8 0x0000000000f4f194 in plugin_initialize (plugin=0x7fffd23fbe30) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_plugin.cc:1163
#9 0x0000000000f4fc1f in plugin_init (argc=0x27784b0 <remaining_argc>, argv=0x7ffff55de860, flags=0) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_plugin.cc:1465
#10 0x0000000000d2b338 in init_server_components () at /ssd/percona/myrocks/facebook-mysql-5.6/sql/mysqld.cc:5853
#11 0x0000000000d2e516 in mysqld_main (argc=17, argv=0x7ffff55de860) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/mysqld.cc:7198
#12 0x0000000000d202dd in main (argc=3, argv=0x7fffffffd958) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/main.cc:25

This is reproduced right on the 5.6.35 HEAD commit 7e78afc but is also pretty old as it was found on PS 5.7.19-17 which was merged up until around April 2017 and on a fresh/empty data set.

[hermanlee]

Both rocksdb_persistent_cache_size and rocksdb_persistent_cache_path needs to be set at startup for this option to work correctly. The assertion happens in debug builds, but the release builds silently fail and the server still boots up. There needs to be a check on startup to fail release builds too.

[hermanlee]

da08d86 fixes the release build for detecting the failure for invalid arguments. For debug builds, it will still assert.

Is it an issue asserting in debug builds for invalid arguments?

[george-lorch]

It looks like the debug assertion is down in rocksdb, so I think since we probably don't want to be changing rocksdb then we will have to live with it, but it is not really a great idea to have something asserting on just a bad combination of options. The main thing is that we need to ensure that any mtr test pass for both debug and release builds.

[george-lorch]

Can we ask the rocksdb team if this assertion is really necessary and useful? Maybe they can remove it and we can live with it until MyRocks catches up?

[hermanlee]

@IslamAbdelRahman thoughts on the assertion in persistent cache when options are invalid?