Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gc-compaction: better job split heuristics #10213

Closed
Tracked by #9114
skyzh opened this issue Dec 19, 2024 · 0 comments · Fixed by #10727
Closed
Tracked by #9114

gc-compaction: better job split heuristics #10213

skyzh opened this issue Dec 19, 2024 · 0 comments · Fixed by #10727
Assignees
@skyzh
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug triaged bugs that were already triaged

Comments

@skyzh
Copy link
Member

skyzh commented Dec 19, 2024 •
edited
Loading

The final stats for downloading is:

INFO compaction_loop{tenant_id=4d77bcd6107af3559d33545edb82b215 shard_id=0308}:run:scheduled_compact_timeline{timeline_id=d245f43c62cd81fc104e168b6786c110}: finish downloading layers, downloaded=14114111488, total=28083863552, ratio=0.50

While the estimation is:

INFO compaction_loop{tenant_id=4d77bcd6107af3559d33545edb82b215 shard_id=0308}:run: splitting compaction job: 000000000000000000000000000000000000..000000067F00004005000080000000D78000, estimated_size=4452556800

4GB vs 14GB

This caused each job is running super slow and spent a lot of time downloading files.
@skyzh skyzh added c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug labels Dec 19, 2024
@skyzh skyzh self-assigned this Dec 19, 2024
@skyzh skyzh mentioned this issue Dec 20, 2024
Open
28 tasks
@jcsp jcsp added the triaged bugs that were already triaged label Jan 9, 2025
@skyzh skyzh mentioned this issue Feb 7, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Feb 10, 2025
@skyzh
feat(pageserver): better gc_compaction_split heuristics (#10727)
b0c7ee0 
## Problem

close #10213

`range_search` only returns the top-most layers that may satisfy the
search, so it doesn't include all layers that might be accessed (the
user needs to recursively call this function). We need to retrieve the
full layer map and find overlaps in order to have a correct heuristics
of the job split.

## Summary of changes

Retrieve all layers and find overlaps instead of doing `range_search`.
The patch also reduces the time holding the layer map read guard.

Signed-off-by: Alex Chi Z <chi@neon.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@skyzh skyzh

Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug triaged bugs that were already triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(pageserver): better gc_compaction_split heuristics neondatabase/neon
2 participants
@jcsp @skyzh