db: improve visibility into snapshots #1204
Labels
Comments
4 tasks
This was referenced Jul 14, 2022
|
We have marked this issue as stale because it has been inactive for |
jbowens
added a commit
to jbowens/cockroach
that referenced
this issue
Apr 3, 2023
Expose two new storage-engine metrics that provide visibility into LSM snapshots. These two new metrics record cumulative count and size of obsolete keys written to sstables during flushes and compactions. Epic: None Release note (ops change): Expose two new timeseries metrics `storage.compactions.keys.pinned.count` and `storage.compactions.keys.pinned.bytes` providing some observability into the volume of keys preserved by open LSM snapshots. Informs cockroachdb/pebble#1204.
jbowens
added a commit
to jbowens/cockroach
that referenced
this issue
Apr 5, 2023
Expose two new storage-engine metrics that provide visibility into LSM snapshots. These two new metrics record cumulative count and size of obsolete keys written to sstables during flushes and compactions. Epic: None Release note (ops change): Expose two new timeseries metrics `storage.compactions.keys.pinned.count` and `storage.compactions.keys.pinned.bytes` providing some observability into the volume of keys preserved by open LSM snapshots. Informs cockroachdb/pebble#1204.
craig bot
pushed a commit
to cockroachdb/cockroach
that referenced
this issue
Apr 6, 2023
100500: sql/schemachanger: extend DML injection tests r=fqazi a=fqazi These changes address the following things in the declarative schema changer 1. Extend DML injection testing into more schema changes in our supported set of changes 2. Update rollback testing to include DML for both happy/unhappy paths (this allows for more coverage). 3. Address the constraint name not being properly populated in certain error cases while a schema change is active with DML. 100518: kvserver: expose LSM snapshot storage-engine timeseries metrics r=RaduBerinde a=jbowens Expose two new storage-engine metrics that provide visibility into LSM snapshots. These two new metrics record cumulative count and size of obsolete keys written to sstables during flushes and compactions. Epic: None Release note (ops change): Expose two new timeseries metrics `storage.compactions.keys.pinned.count` and `storage.compactions.keys.pinned.bytes` providing some observability into the volume of keys preserved by open LSM snapshots. Informs cockroachdb/pebble#1204. Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com> Co-authored-by: Jackson Owens <jackson@cockroachlabs.com>
jbowens
added a commit
to jbowens/cockroach
that referenced
this issue
Apr 6, 2023
Expose two new storage-engine metrics that provide visibility into LSM snapshots. These two new metrics record cumulative count and size of obsolete keys written to sstables during flushes and compactions. Epic: None Release note (ops change): Expose two new timeseries metrics `storage.compactions.keys.pinned.count` and `storage.compactions.keys.pinned.bytes` providing some observability into the volume of keys preserved by open LSM snapshots. Informs cockroachdb/pebble#1204.
|
We have marked this issue as stale because it has been inactive for |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Open snapshots can impact performance. During compactions, they increase write amplification by requiring us to write keys that we otherwise could elide. During iteration, iterators must sift through obsolete keys preserved by snapshots. The compensated size compaction heuristic doesn't take snapshots into account (see #872), which can exacerbate the above issues by prioritizing compactions that must write obsolete keys.
We have some ideas (eg, #847) in lessening the impact of snapshots. It's hard to prioritize any work here without understanding how much of an impact snapshots have on real workloads. Some ideas for improvising visibility here:
obsolete.keys.countproperty to sstables, counting the total number of keys within an sstable that could have been elided (from range deletions, point deletions, overwriting, etc) if there were no open snapshots.obsolete.keys.sizeproperty to sstables, summing the raw key size and raw value size of the above.manifest.Annotatorthat sums the above sstable properties over an entire level and expose onpebble.Metrics.pebble.Metrics.Jira issue: PEBBLE-208
The text was updated successfully, but these errors were encountered: