use bloom filter, add event listener to collect stats of compaction, improve update rocksdb options #1959

critical27 · 2020-03-24T07:24:02Z

Use bloom filter by default
Add some options to open partitioned index/filter
Add a EventListener to collect stats of compaction.
Improve udpate rocksdb configs, for those options which can't change dynamically, it would take effect after reboot.

@Amber1990Zhang, would you help to update the doc configs-syntax, because some options is not supported anymore, thx.

close #1966

darionyaphet · 2020-03-24T07:33:30Z

src/kvstore/EventListner.h

+namespace nebula {
+namespace kvstore {
+
+class EventListener : public rocksdb::EventListener {


darionyaphet · 2020-03-24T07:50:23Z

src/kvstore/raftex/RaftPart.cpp

@@ -28,7 +28,7 @@ DEFINE_uint64(raft_snapshot_timeout, 60 * 5, "Max seconds between two snapshot r

 DEFINE_uint32(max_batch_size, 256, "The max number of logs in a batch");

-DEFINE_int32(wal_ttl, 86400, "Default wal ttl");
+DEFINE_int32(wal_ttl, 14400, "Default wal ttl");


current is 4 hour ?

Current is 1 day, @whitewum suggest change it shorter.

whitewum · 2020-03-25T03:29:51Z

src/kvstore/RocksEngineConfig.cpp

@@ -87,6 +91,15 @@ rocksdb::Status initRocksdbOptions(rocksdb::Options &baseOpts) {
 static std::shared_ptr<rocksdb::Cache> blockCache
 = rocksdb::NewLRUCache(FLAGS_rocksdb_block_cache * 1024 * 1024);
 bbtOpts.block_cache = blockCache;
+ bbtOpts.filter_policy.reset(rocksdb::NewBloomFilterPolicy(10, false));
+ if (FLAGS_enable_partitioned_index_filter) {


how about add prefix bloom filter? for the prefix 12bytes

Good suggestion. Later I will check if it improves perf.

…s of compaction

bright-starry-sky · 2020-03-25T13:03:26Z

src/meta/processors/configMan/SetConfigProcessor.cpp

@@ -14,7 +14,7 @@ std::unordered_set<std::string> SetConfigProcessor::mutableFields_ = {
 // rocksdb_column_family_options
 "disable_auto_compactions",
 // TODO: write_buffer_size will cause rocksdb crash
- // "write_buffer_size",
+ "write_buffer_size",


"write_buffer_size" works well now ? if yes , do we need to uncomment in MetaServerBasedPartManager::onSpaceOptionUpdated?

"write_buffer_size" works well now ? if yes , do we need to uncomment in MetaServerBasedPartManager::onSpaceOptionUpdated?

Sorry, the comment has been removed. I'm wrong.

Forgot to remove the TODO : )

bright-starry-sky

Excellent work, I learned some new skills.
LGTM.

bright-starry-sky · 2020-03-25T13:14:53Z

src/common/base/Configuration.cpp

@@ -160,7 +164,7 @@ Status Configuration::fetchAsSubConf(const char *key, Configuration &subconf) co
 }


-Status Configuration::updateStringField(const char* key, const std::string& val) {
+Status Configuration::upsertStringField(const char* key, const std::string& val) {


A question, Why naming upsert?
What I understand that upsert is : should be override the old value if key exists; should be insert a new value if the key doesn't exist.

Exactly, so we can insert some option which doesn't exists, and update some option with a new value. The function actually is a upsert, a bad name.

dangleptr · 2020-03-26T02:29:57Z

src/kvstore/PartManager.cpp

 "level0_file_num_compaction_trigger",
 "level0_slowdown_writes_trigger",
 "level0_stop_writes_trigger",
 "target_file_size_base",
 "target_file_size_multiplier",
 "max_bytes_for_level_base",
 "max_bytes_for_level_multiplier",
- "ttl",
- "block_size",
- "block_restart_interval"
 };
 static std::unordered_set<std::string> supportedDbOpt = {
 "max_total_wal_size",
 "delete_obsolete_files_period_micros",
 "max_background_jobs",


support max_subcompactions

it can't be changed dynamically, for those options, it would take effect when reboot.

dangleptr

Well done. LGTM.
Thanks for taking care of it.

darionyaphet

Great Work

@yixinlu

* Upgrade rocksdb to 6.7.3 (#1948) Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: yaphet <darion.wang@vesoft.com> * Detect OS type and exit if not built in Linux (#1951) * Detect OS type and exit if not built in Linux Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> * Disable for FreeBSD Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> * [doc] user-permission-mng (#1929) * alter user & change psw * built in roles * cbs comments * fix comments * update table * addenable_authorize doc * [doc] ttl doc modification (#1939) * ttl doc modification * fix comments * yt comments * silver days comments * timestamp update * Let action not fail fast. (#1909) * Let action not fail fast. * Disable the ci will product artifact. * Remove removePrefix interface in kvstore (#1953) Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: yaphet <darion.wang@vesoft.com> * using admin client in MetaServiceHandler (#1864) * [doc] fix typo (#1954) * [doc]show statements (#1924) * [doc]customized dir (#1906) * Fixed a bug caused by not filtering the versions of edges (#1962) * Fixed a bug caused by not filtering the versions of edges when executing "go from vid over *". * address CPWstatic's comment. * Modify the listEdges interface and add parameters to directly return all latest versions of edges. * address laura's comment. Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * [doc]modify job manger doc (#1957) * modify job manger doc * add flush Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Fix delete verticse (#1970) * Fix the read performance issue after delete range with rocksdb-5.15.10 (#1973) * Fix the read performace issue after delete range with rocksdb-5.15.10 * Fix the move issue * fix the move issue * Rebase and address bright-starry-sky's comments Co-authored-by: heng <heng.chen@vesoft.com> * Remove the LOG(FATAL) when encounting the unknown log type (#1978) * 1,Improve customer friendliness; 2, fixed typo error (#1980) * upgrade spark download address (#1981) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * fix-typo-in-doc (#1976) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * modify commands in build doc (#1969) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * modify log level (#1945) Co-authored-by: dangleptr <37216992+dangleptr@users.noreply.github.com> Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * use bloom filter, add event listener to collect stats of compaction, improve update rocksdb options (#1959) * open bloom filter in default conf, add event listener to collect stats of compaction * imporve update rocksdb configs * fix ut error Co-authored-by: yaphet <darion.wang@vesoft.com> * Update the interface atomicOp to support empty string (#2002) Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * update to rc4 (#1968) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Fixed if you execute "go from xx over *", and when there is no edge under a space, (#1983) an unreasonable error message is returned (SpaceId `xxx 'is nonexistent) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * [doc]pr-commit-msg (#1993) * pr-commit-msg * add examples Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * doc:modify ttl query (#2006) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Let merge coverage report faster by fastcov instead of lcov. (#1943) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Auto check keyword (#1922) * Supports the detection keyword, name_label supports string * add tests * address Shylock's comment * modify to add reserved keyword Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Fix coredump for update if encountering bad format row (#2024) Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: bright-starry-sky <56461666+bright-starry-sky@users.noreply.github.com> * fixed loop depend problem for compile link (#1995) * fixed loop depend problem * fixed error for clang9 and gcc9 * Addressed laura-ding's comment * Check all reader is null or not before using it (#2029) * fix go without yield return errror result (#2012) * fix go without yield return errror result * fix format * address dangleptr's comment * address issue change TTL_DURATION not negative * add onlyStructure is true test case Co-authored-by: bright-starry-sky <56461666+bright-starry-sky@users.noreply.github.com> * fixed bug for delete vertex without edge (#2001) * When deleting a vertex, if the edge does not exist, then we should continue to delete the vertex instead of reporting an error. * added the testcase for delete vertex without edge. * add alias for lookup (#2000) * fix bug 1990/1991 (#2007) * 1. meta client sub mit job only to leader 2. sort jobs in desc id order * now show jobs return in desc order, adjust UT Co-authored-by: dangleptr <37216992+dangleptr@users.noreply.github.com> * upload package to oss (#1972) * upload oss * update * Remove redundant * address @yixinlu's comment Co-authored-by: dutor <440396+dutor@users.noreply.github.com> Co-authored-by: dangleptr <37216992+dangleptr@users.noreply.github.com> Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: yaphet <darion.wang@vesoft.com> Co-authored-by: JaySon <jayson.hjs@gmail.com> Co-authored-by: Amber Zhang <42762957+Amber1990Zhang@users.noreply.github.com> Co-authored-by: Shylock Hg <33566796+Shylock-Hg@users.noreply.github.com> Co-authored-by: Simon Liu <331435+monadbobo@users.noreply.github.com> Co-authored-by: dutor <440396+dutor@users.noreply.github.com> Co-authored-by: bright-starry-sky <56461666+bright-starry-sky@users.noreply.github.com> Co-authored-by: Liuxue-Yang <50105172+Liuxue-Yang@users.noreply.github.com> Co-authored-by: laura-ding <48548375+laura-ding@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com> Co-authored-by: liuyu85cn <52276794+liuyu85cn@users.noreply.github.com>

…improve update rocksdb options (vesoft-inc#1959) * open bloom filter in default conf, add event listener to collect stats of compaction * imporve update rocksdb configs * fix ut error Co-authored-by: yaphet <darion.wang@vesoft.com>

critical27 requested review from darionyaphet, CPWstatic, dangleptr, laura-ding, liuyu85cn, bright-starry-sky and panda-sheep March 24, 2020 07:24

darionyaphet reviewed Mar 24, 2020

View reviewed changes

whitewum reviewed Mar 25, 2020

View reviewed changes

open bloom filter in default conf, add event listener to collect stat…

d7258ea

…s of compaction

critical27 force-pushed the conf branch from 0971769 to d7258ea Compare March 25, 2020 10:06

imporve update rocksdb configs

f8cca45

critical27 added the ready-for-testing PR: ready for the CI test label Mar 25, 2020

critical27 changed the title ~~use bloom filter, add event listener to collect stats of compaction~~ use bloom filter, add event listener to collect stats of compaction, improve update rocksdb options Mar 25, 2020

bright-starry-sky reviewed Mar 25, 2020

View reviewed changes

dangleptr reviewed Mar 26, 2020

View reviewed changes

fix ut error

c8f7391

critical27 requested a review from amber-moe March 26, 2020 03:44

dangleptr approved these changes Mar 26, 2020

View reviewed changes

darionyaphet approved these changes Mar 27, 2020

View reviewed changes

Merge branch 'master' into conf

7582d23

whitewum added this to the R201910_RC4 milestone Mar 27, 2020

dangleptr merged commit 249b46c into vesoft-inc:master Mar 27, 2020

dangleptr mentioned this pull request Mar 30, 2020

change default wal ttl from 1 day to 4 hours. #1947

Closed

amber-moe self-assigned this Mar 31, 2020

critical27 deleted the conf branch May 14, 2020 08:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use bloom filter, add event listener to collect stats of compaction, improve update rocksdb options #1959

use bloom filter, add event listener to collect stats of compaction, improve update rocksdb options #1959

critical27 commented Mar 24, 2020 •

edited

Loading

darionyaphet Mar 24, 2020

darionyaphet Mar 24, 2020

critical27 Mar 24, 2020

whitewum Mar 25, 2020

critical27 Mar 25, 2020

bright-starry-sky Mar 25, 2020

bright-starry-sky Mar 25, 2020

critical27 Mar 26, 2020

bright-starry-sky left a comment

bright-starry-sky Mar 25, 2020

critical27 Mar 26, 2020

dangleptr Mar 26, 2020 •

edited

Loading

critical27 Mar 26, 2020

dangleptr left a comment

darionyaphet left a comment

use bloom filter, add event listener to collect stats of compaction, improve update rocksdb options #1959

use bloom filter, add event listener to collect stats of compaction, improve update rocksdb options #1959

Conversation

critical27 commented Mar 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bright-starry-sky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dangleptr Mar 26, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dangleptr left a comment

Choose a reason for hiding this comment

darionyaphet left a comment

Choose a reason for hiding this comment

critical27 commented Mar 24, 2020 •

edited

Loading

dangleptr Mar 26, 2020 •

edited

Loading