kv/kvserver: split incoming snapshot user keys into multiple sstables #67284
Assignees
Labels
A-storage
Relating to our storage engine (Pebble) on-disk storage.
branch-release-23.2.15-rc
branch-release-24.1
Used to mark GA and release blockers, technical advisories, and bugs for 24.1
branch-release-24.2
Used to mark GA and release blockers, technical advisories, and bugs for 24.2
C-performance
Perf of queries or internals. Solution not expected to change functional behavior.
T-kv
KV Team
Comments
jbowens
added
C-performance
|
I believe the only change required is to |
|
Ack, I can make the change myself, but I probably won't get it for a little while. |
raggar
added a commit
to raggar/cockroach
that referenced
this issue
Jun 7, 2023
We ingest a fixed number of sstables, corresponding to the range's various contiguous keyspaces. When the default range size increased from 64 MB to 512 MB, we started ingesting larger user data sstables which negatively impacts compation. This change creates a new sstable if the current one is too large (greater than 128MiB). When the default range size increased from 64 MB to 512 MB, we started ingesting user data sstables up to 512 MB. Release note: None Fixes: cockroachdb#67284
raggar
added a commit
to raggar/cockroach
that referenced
this issue
Jul 13, 2023
This pr consists of two main changes. The first is moving the `MultiSSTWriter` to package `storage`. The second change comes from the following motivation. Currently, when receiving a snapshot, we ingest a fixed number of sstables, corresponding to the range's various contiguous keyspaces. When the default range size increased from 64 MB to 512 MB, we started ingesting user data sstables up to 512 MB. These large files cause more expensive compactions. As a result of limiting the size of SSTables, this pr adds range key buffering and truncation to the existing MultiSSTWriter logic. Release note: None Informs: cockroachdb#67284
craig bot
pushed a commit
that referenced
this issue
Aug 14, 2024
127997: kvserver: split snapshot SSTables for mvcc keys into multiple SSTs r=jbowens a=itsbilal Previously, we'd only create one sstable for all mvcc keys in a range when ingesting a rebalance/recovery snapshot into Pebble. This increased write-amp in Pebble as more sstables would have to be compacted into it (or the sstable then split into smaller ones in Pebble), and had other consequences such as massive filter blocks in the large singular sstable. This change adds a new cluster setting, kv.snapshot_rebalance.max_sst_size, that sets the max size of the sstables containing user/mvcc keys in a range. If an sstable exceeds this size in multiSSTWriter, we roll over that sstable and create a new one. Epic: CRDB-8471 Fixes: #67284 Release note (performance improvement): Reduce the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble. Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>
github-project-automation
bot
moved this from In Progress (this milestone)
to Done
in Storage
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
Based on the specified backports for linked PR #127997, I applied the following new label(s) to this issue: branch-release-24.1, branch-release-24.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
blathers-crl
bot
added
branch-release-24.1
blathers-crl bot
pushed a commit
that referenced
this issue
Aug 14, 2024
Previously, we'd only create one sstable for all mvcc keys in a range when ingesting a rebalance/recovery snapshot into Pebble. This increased write-amp in Pebble as more sstables would have to be compacted into it (or the sstable then split into smaller ones in Pebble), and had other consequences such as massive filter blocks in the large singular sstable. This change adds a new cluster setting, kv.snapshot_rebalance.max_sst_size, that sets the max size of the sstables containing user/mvcc keys in a range. If an sstable exceeds this size in multiSSTWriter, we roll over that sstable and create a new one. Epic: CRDB-8471 Fixes: #67284 Release note (performance improvement): Reduce the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble.
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Aug 14, 2024
Previously, we'd only create one sstable for all mvcc keys in a range when ingesting a rebalance/recovery snapshot into Pebble. This increased write-amp in Pebble as more sstables would have to be compacted into it (or the sstable then split into smaller ones in Pebble), and had other consequences such as massive filter blocks in the large singular sstable. This change adds a new cluster setting, kv.snapshot_rebalance.max_sst_size, that sets the max size of the sstables containing user/mvcc keys in a range. If an sstable exceeds this size in multiSSTWriter, we roll over that sstable and create a new one. Epic: CRDB-8471 Fixes: cockroachdb#67284 Release note (performance improvement): Reduce the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Aug 14, 2024
This change exports the truncateAndFlush method in keyspan.Fragmenter. Necessary to unblock cockroachdb/cockroach#67284 .
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Aug 14, 2024
This change exports the truncateAndFlush method in keyspan.Fragmenter. Necessary to unblock cockroachdb/cockroach#67284 .
itsbilal
added a commit
to cockroachdb/pebble
that referenced
this issue
Aug 15, 2024
This change exports the truncateAndFlush method in keyspan.Fragmenter. Necessary to unblock cockroachdb/cockroach#67284 .
itsbilal
added a commit
to cockroachdb/pebble
that referenced
this issue
Aug 15, 2024
This change exports the truncateAndFlush method in keyspan.Fragmenter. Necessary to unblock cockroachdb/cockroach#67284 .
itsbilal
added a commit
that referenced
this issue
Sep 18, 2024
Previously, we'd only create one sstable for all mvcc keys in a range when ingesting a rebalance/recovery snapshot into Pebble. This increased write-amp in Pebble as more sstables would have to be compacted into it (or the sstable then split into smaller ones in Pebble), and had other consequences such as massive filter blocks in the large singular sstable. This change adds a new cluster setting, kv.snapshot_rebalance.max_sst_size, that sets the max size of the sstables containing user/mvcc keys in a range. If an sstable exceeds this size in multiSSTWriter, we roll over that sstable and create a new one. Epic: CRDB-8471 Fixes: #67284 Release note (performance improvement): Reduce the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble.
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Sep 30, 2024
Previously, we'd only create one sstable for all mvcc keys in a range when ingesting a rebalance/recovery snapshot into Pebble. This increased write-amp in Pebble as more sstables would have to be compacted into it (or the sstable then split into smaller ones in Pebble), and had other consequences such as massive filter blocks in the large singular sstable. This change adds a new cluster setting, kv.snapshot_rebalance.max_sst_size, that sets the max size of the sstables containing user/mvcc keys in a range. If an sstable exceeds this size in multiSSTWriter, we roll over that sstable and create a new one. Epic: CRDB-8471 Fixes: cockroachdb#67284 Release note (performance improvement): Reduce the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Nov 6, 2024
This change exports the truncateAndFlush method in keyspan.Fragmenter. Necessary to unblock cockroachdb/cockroach#67284 .
itsbilal
added a commit
to cockroachdb/pebble
that referenced
this issue
Nov 7, 2024
This change exports the truncateAndFlush method in keyspan.Fragmenter. Necessary to unblock cockroachdb/cockroach#67284 .
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Nov 7, 2024
Previously, we'd only create one sstable for all mvcc keys in a range when ingesting a rebalance/recovery snapshot into Pebble. This increased write-amp in Pebble as more sstables would have to be compacted into it (or the sstable then split into smaller ones in Pebble), and had other consequences such as massive filter blocks in the large singular sstable. This change adds a new cluster setting, kv.snapshot_rebalance.max_sst_size, that sets the max size of the sstables containing user/mvcc keys in a range. If an sstable exceeds this size in multiSSTWriter, we roll over that sstable and create a new one. Epic: CRDB-8471 Fixes: cockroachdb#67284 Release note (performance improvement): Reduce the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble.
|
Based on the specified backports for linked PR #134526, I applied the following new label(s) to this issue: branch-release-23.2.15-rc. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
blathers-crl bot
pushed a commit
that referenced
this issue
Nov 12, 2024
Previously, we'd only create one sstable for all mvcc keys in a range when ingesting a rebalance/recovery snapshot into Pebble. This increased write-amp in Pebble as more sstables would have to be compacted into it (or the sstable then split into smaller ones in Pebble), and had other consequences such as massive filter blocks in the large singular sstable. This change adds a new cluster setting, kv.snapshot_rebalance.max_sst_size, that sets the max size of the sstables containing user/mvcc keys in a range. If an sstable exceeds this size in multiSSTWriter, we roll over that sstable and create a new one. Epic: CRDB-8471 Fixes: #67284 Release note (performance improvement): Reduce the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble.
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Nov 13, 2024
Previously, we'd only create one sstable for all mvcc keys in a range when ingesting a rebalance/recovery snapshot into Pebble. This increased write-amp in Pebble as more sstables would have to be compacted into it (or the sstable then split into smaller ones in Pebble), and had other consequences such as massive filter blocks in the large singular sstable. This change adds a new cluster setting, kv.snapshot_rebalance.max_sst_size, that sets the max size of the sstables containing user/mvcc keys in a range. If an sstable exceeds this size in multiSSTWriter, we roll over that sstable and create a new one. Epic: CRDB-8471 Fixes: cockroachdb#67284 Release note (performance improvement): Reduce the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, when receiving a snapshot, we ingest a fixed number of sstables, corresponding to the range's various contiguous keyspaces. When the default range size increased from 64 MB to 512 MB, we started ingesting user data sstables up to 512 MB. These large files cause more expensive compactions.
Here's a relevant TODO.
Related thread: https://cockroachlabs.slack.com/archives/CAC6K3SLU/p1686237663708959?thread_ts=1686084258.076069&cid=CAC6K3SLU
Related to cockroachdb/pebble#1181.
cc @sumeerbhola
Jira issue: CRDB-8471
The text was updated successfully, but these errors were encountered: