Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

db: IngestOperationStats absent for flushable ingests #2421

Closed
jbowens opened this issue Mar 30, 2023 · 1 comment · Fixed by #2425
Closed

db: IngestOperationStats absent for flushable ingests #2421

jbowens opened this issue Mar 30, 2023 · 1 comment · Fixed by #2425
Labels
A-storage T-storage

Comments

@jbowens
Copy link
Collaborator

jbowens commented Mar 30, 2023

We should calculate approximate stats to accommodate admission control.

pebble/ingest.go

Lines 661 to 663 in 2e9d3f5

// Currently, this value is completely accurate, but we are allowing this to
// be approximate once https://github.com/cockroachdb/pebble/issues/25 is
// implemented.

@sumeerbhola
Copy link
Collaborator

Glad you noticed this! I think it would be an ok approximation if in ingestMemtableOverlaps we computed the full list of files that overlapped with the memtable, and assumed that they will all go into L0 when later doing runIngestFlush.

@jbowens jbowens mentioned this issue Mar 30, 2023
Closed
jbowens added a commit to jbowens/pebble that referenced this issue Mar 31, 2023
@jbowens
db: populate return statistics for flushable ingests
853ec5e 
The IngestWithStats ingestion entrypoint returnrs statistics about the ingest.
Cockroach uses these statistics to inform admission control. During a flushable
ingest, at commit time the number of bytes that will be ingested into L0 is
unknown. This commit adds an approximation based on which tables overlapped the
flushable queue. This required adjusting the memtable overlap logic to avoid
short circuiting as soon as overlap is found.

This approximation is rough and may be improved with future work, such as cockroachdb#2112.

Close cockroachdb#2421.
Informs cockroachdb#2112.
@jbowens jbowens mentioned this issue Mar 31, 2023
Merged
jbowens added a commit to jbowens/pebble that referenced this issue Apr 3, 2023
@jbowens
db: populate return statistics for flushable ingests
e39edd4 
The IngestWithStats ingestion entrypoint returnrs statistics about the ingest.
Cockroach uses these statistics to inform admission control. During a flushable
ingest, at commit time the number of bytes that will be ingested into L0 is
unknown. This commit adds an approximation based on which tables overlapped the
flushable queue. This required adjusting the memtable overlap logic to avoid
short circuiting as soon as overlap is found.

This approximation is rough and may be improved with future work, such as cockroachdb#2112.

Close cockroachdb#2421.
Informs cockroachdb#2112.
jbowens added a commit that referenced this issue Apr 3, 2023
@jbowens
db: populate return statistics for flushable ingests
b84a7ec 
The IngestWithStats ingestion entrypoint returnrs statistics about the ingest.
Cockroach uses these statistics to inform admission control. During a flushable
ingest, at commit time the number of bytes that will be ingested into L0 is
unknown. This commit adds an approximation based on which tables overlapped the
flushable queue. This required adjusting the memtable overlap logic to avoid
short circuiting as soon as overlap is found.

This approximation is rough and may be improved with future work, such as #2112.

Close #2421.
Informs #2112.
jbowens added a commit to jbowens/pebble that referenced this issue Apr 3, 2023
@jbowens
db: populate return statistics for flushable ingests
be3052b 
The IngestWithStats ingestion entrypoint returnrs statistics about the ingest.
Cockroach uses these statistics to inform admission control. During a flushable
ingest, at commit time the number of bytes that will be ingested into L0 is
unknown. This commit adds an approximation based on which tables overlapped the
flushable queue. This required adjusting the memtable overlap logic to avoid
short circuiting as soon as overlap is found.

This approximation is rough and may be improved with future work, such as cockroachdb#2112.

Close cockroachdb#2421.
Informs cockroachdb#2112.
@jbowens jbowens mentioned this issue Apr 3, 2023
Merged
jbowens added a commit that referenced this issue Apr 3, 2023
@jbowens
db: populate return statistics for flushable ingests
6de4e6c 
The IngestWithStats ingestion entrypoint returnrs statistics about the ingest.
Cockroach uses these statistics to inform admission control. During a flushable
ingest, at commit time the number of bytes that will be ingested into L0 is
unknown. This commit adds an approximation based on which tables overlapped the
flushable queue. This required adjusting the memtable overlap logic to avoid
short circuiting as soon as overlap is found.

This approximation is rough and may be improved with future work, such as #2112.

Close #2421.
Informs #2112.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-storage T-storage
Projects
Storage
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

db: populate return statistics for flushable ingests jbowens/pebble
2 participants
@jbowens @sumeerbhola