Disk utilization fixes #9225
Merged
Disk utilization fixes #9225
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This changes the approach to adjusting the amount of concurrency used for snapshotting to be based on the snapshot latency vs cardinality. The cardinality approach could use too much concurrency and increase the number of level 1 TSM files too quickly which incurs more disk IO. The latency model seems to adjust better to different workloads.
This reverts commit 171b427.
ghost
assigned 
jwilder
force-pushed
the
jw-snapshost-concurrency
branch
6 times, most recently
from
ebb518d to
9b8929f
Compare
This limits the disk IO for writing TSM files during compactions and snapshots. This helps reduce the spiky IO patterns on SSDs and when compactions run very quickly.
jwilder
force-pushed
the
jw-snapshost-concurrency
branch
2 times, most recently
from
e321ee9 to
5893d49
Compare
Increase level 1 min criteria, fix only fast compactions getting run, and fix very large generations getting included in optimize plans.
jwilder
force-pushed
the
jw-snapshost-concurrency
branch
from
5893d49 to
2d85ff1
Compare
jwilder
added a commit
that referenced
this pull request
Dec 15, 2017
e-dard
added a commit
that referenced
this pull request
Jul 18, 2018
PR #9204 introduced a maximum default concurrent compaction limit of 4. The idea was to reduce IO utilisation on large systems with many cores, and high write load. Often on these systems, disks were not scaled appropriately to to the write volume, and while the write path could keep up, compactions would saturate disks. In #9225 work was done to reduce IO saturation by limiting the compaction throughput. To some extent, both #9204 and #9225 work towards solving the same problem. We have recently begun to notice larger clusters to suffer from situations where compactions are not keeping up because they have been scaled up, but the limit of 4 has stayed in place. While users can manually override the setting, it seems more user friendly if we remove the limit by default, and set it manually in cases where compactions are causing too much IO on large boxes.
e-dard
added a commit
that referenced
this pull request
Jul 18, 2018
PR #9204 introduced a maximum default concurrent compaction limit of 4. The idea was to reduce IO utilisation on large systems with many cores, and high write load. Often on these systems, disks were not scaled appropriately to to the write volume, and while the write path could keep up, compactions would saturate disks. In #9225 work was done to reduce IO saturation by limiting the compaction throughput. To some extent, both #9204 and #9225 work towards solving the same problem. We have recently begun to notice larger clusters to suffer from situations where compactions are not keeping up because they have been scaled up, but the limit of 4 has stayed in place. While users can manually override the setting, it seems more user friendly if we remove the limit by default, and set it manually in cases where compactions are causing too much IO on large boxes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Required for all non-trivial PRs
This is a followup to #9206 to fix the higher disk utilization and lower write throughput in 1.4.2. One change in #9206 was to increase the snapshot size in order to create fewer and larger level 1 TSM files. This helped to slow down the frequency of compactions. Unfortunately, users with many databases or lots of active shards could OOM the process each shard is consuming more memory now.
This PR reverts that fix and fixes the issue slightly differently.
This is a run on c4.4xlarge w/ 10B values, 2.5m series and 5 writers. The 1.4.2 run is only 1/5 of the way through, but the performance difference can be seen already. You can see in 1.4.2 that there are too many compactions getting run (combined w/ other issues) which increases disk utilization and lowers write throughput. This PR has perf that is similar or better to 1.3 in different areas. There are significantly fewer compactions, and write throughput is ~30% better than 1.3 now.
Fixes #9217 #9201