Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

db: wire virtual sstable space amp data structure in the compaction picker #2323

Closed
bananabrick opened this issue Feb 13, 2023 · 1 comment · Fixed by #3370
Closed

db: wire virtual sstable space amp data structure in the compaction picker #2323

bananabrick opened this issue Feb 13, 2023 · 1 comment · Fixed by #3370
Assignees
@sumeerbhola
Labels
A-storage T-storage

Comments

@bananabrick
Copy link
Contributor

bananabrick commented Feb 13, 2023 •
edited
Loading

See the details here for how the VirtualizedSize field should be used for compaction picking.
@bananabrick bananabrick mentioned this issue Feb 16, 2023
Closed
8 tasks
@bananabrick
Copy link
Contributor Author

Datastructure changes for this are present in #2352.

@bananabrick bananabrick changed the title db: test and implement compaction picking heuristics for virtual sstables db: wire virtual sstable space amp data structure in the compaction picker Mar 28, 2023
@bananabrick bananabrick removed their assignment May 23, 2023
@bananabrick bananabrick mentioned this issue Sep 18, 2023
Closed
@bananabrick bananabrick self-assigned this Sep 18, 2023
@bananabrick bananabrick mentioned this issue Sep 18, 2023
Closed
@nicktrav nicktrav assigned sumeerbhola and unassigned bananabrick Oct 18, 2023
sumeerbhola added a commit to sumeerbhola/pebble that referenced this issue Mar 1, 2024
@sumeerbhola
db: account for garbage in backing files when picking a compaction
da86dfe 
When we consider a virtual sst as a candidate for the seed file for a
compaction, the size of that sst is adjusted by a fraction of the
garbage in the backing sst, where the fraction is the reciprocal of
the number of virtual ssts that are referencing this backing sst.

The objective is to reduce space amp, by being aware of garbage
accumulation, when picking compactions.

Fixes cockroachdb#2323
@sumeerbhola sumeerbhola mentioned this issue Mar 1, 2024
Merged
sumeerbhola added a commit to sumeerbhola/pebble that referenced this issue Mar 3, 2024
@sumeerbhola
db: account for garbage in backing files when picking a compaction
3f4d30d 
When we consider a virtual sst as a candidate for the seed file for a
compaction, the size of that sst is adjusted by a fraction of the
garbage in the backing sst, where the fraction is the reciprocal of
the number of virtual ssts that are referencing this backing sst.

The objective is to reduce space amp, by being aware of garbage
accumulation, when picking compactions.

Fixes cockroachdb#2323
sumeerbhola added a commit that referenced this issue Mar 5, 2024
@sumeerbhola
db: account for garbage in backing files when picking a compaction
1ff6ed8 
When we consider a virtual sst as a candidate for the seed file for a
compaction, the size of that sst is adjusted by a fraction of the
garbage in the backing sst, where the fraction is the reciprocal of
the number of virtual ssts that are referencing this backing sst.

The objective is to reduce space amp, by being aware of garbage
accumulation, when picking compactions.

Fixes #2323
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sumeerbhola sumeerbhola

Labels
A-storage T-storage
Projects
Storage
Archived in project
Milestone
No milestone
2 participants
@bananabrick @sumeerbhola