crl-release-23.1: db: populate return statistics for flushable ingests #2429
Conversation
The IngestWithStats ingestion entrypoint returnrs statistics about the ingest. Cockroach uses these statistics to inform admission control. During a flushable ingest, at commit time the number of bytes that will be ingested into L0 is unknown. This commit adds an approximation based on which tables overlapped the flushable queue. This required adjusting the memtable overlap logic to avoid short circuiting as soon as overlap is found. This approximation is rough and may be improved with future work, such as cockroachdb#2112. Close cockroachdb#2421. Informs cockroachdb#2112.
|
This change is |
There was a problem hiding this comment.
Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion (waiting on @jbowens and @sumeerbhola)
ingest.go line 1003 at r1 (raw file):
// just memtables. if metaFlushableOverlaps[i] { stats.ApproxIngestedIntoL0Bytes += f.Size
Could we update this during the flush? Or do we want to avoid any delay in the update of this statistic?
There was a problem hiding this comment.
Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion (waiting on @bananabrick and @sumeerbhola)
ingest.go line 1003 at r1 (raw file):
Previously, bananabrick (Arjun Nair) wrote…
Could we update this during the flush? Or do we want to avoid any delay in the update of this statistic?
This stats struct is returned directly to the caller before the flush occurs.
There was a problem hiding this comment.
Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion (waiting on @sumeerbhola)
|
TFTR! |
23.1 backport of #2425.
The IngestWithStats ingestion entrypoint returnrs statistics about the ingest. Cockroach uses these statistics to inform admission control. During a flushable ingest, at commit time the number of bytes that will be ingested into L0 is unknown. This commit adds an approximation based on which tables overlapped the flushable queue. This required adjusting the memtable overlap logic to avoid short circuiting as soon as overlap is found.
This approximation is rough and may be improved with future work, such as #2112.
Informs #2421.
Informs #2112.
Informs cockroachdb/cockroach#100149.