Update DBOptions::IncreaseParallelism to use newer background settings #3208
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Options header file recommends using max_background_jobs rather than directly setting max_background_compactions or max_background_flushes. I've personally seen a performance problem where stalls were happening because the one background flushing thread was blocked that was fixed by this change - cockroachdb/cockroach#19699 (comment)
a-robinson
added a commit
to a-robinson/cockroach
that referenced
this pull request
Nov 30, 2017
This depends on also getting our build flags fixed -- `bytes_per_sync` doesn't work without `-DROCKSDB_RANGESYNC_PRESENT` being set. The 512KB option is somewhat arbitrary, but was chosen as the best value for certain configurations by resources such as https://www.youtube.com/watch?v=pvUqbIeoPzM and https://www.percona.com/live/data-performance-conference-2016/sites/default/files/slides/Percona_RocksDB_v1.3.pdf Also, I'm trying to upstream the related `max_background_jobs` change I mentioned in cockroachdb#19436: facebook/rocksdb#3208. If they don't take that upstream, I'll change it here. Helps with cockroachdb#19436. Release note (performance improvement): Smoothen out disk usage under very write heavy workloads by syncing to disk more frequently.
There was a problem hiding this comment.
@ajkr is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
a-robinson
added a commit
to cockroachdb/rocksdb
that referenced
this pull request
Dec 4, 2017
Summary: The Options header file recommends using max_background_jobs rather than directly setting max_background_compactions or max_background_flushes. I've personally seen a performance problem where stalls were happening because the one background flushing thread was blocked that was fixed by this change - cockroachdb/cockroach#19699 (comment) Closes facebook#3208 Differential Revision: D6473178 Pulled By: ajkr fbshipit-source-id: 67c892ceb7b1909d251492640cb15a0f2262b7ed
a-robinson
added a commit
to a-robinson/cockroach
that referenced
this pull request
Dec 4, 2017
facebook/rocksdb#3211 fixes the CMake build flags to enable the `sync_file_range` syscall which is required by the `bytes_per_sync` feature. facebook/rocksdb#3208 modernizes how background compactions/flushes are configured, which helped prevent stalls in some of my testing. cc cockroachdb#19699
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Options header file recommends using max_background_jobs rather than
directly setting max_background_compactions or max_background_flushes.
I've personally seen a performance problem where stalls were happening
because the one background flushing thread was blocked that was fixed
by this change -
cockroachdb/cockroach#19699 (comment)