Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PCP-27: level compaction skip some sst to reduce write amplification #375

Open
zhangjinpeng87 opened this issue Nov 4, 2019 · 4 comments

Comments

@zhangjinpeng87
Copy link
Member

zhangjinpeng87 commented Nov 4, 2019

Description

Ln sst1[a...b e...f]
Ln+1 sst2[a...b] sst3[c...d] sst4[e...f]
Assume a level compaction picks sst1, sst2, sst3 and sst4 as source ssts, we can see sst3's key range is a gap in sst1, but their ranges overlaps. And sst3's content will be unnecessarily re-written in level n+1, this will cause write amplification.
In TiDB the gap between index keys and row keys for the same row is broad, and will cause extra write amplification.

Difficulty

  • Hard

Score

  • 3600

Mentor(s)

Recommended Skills

  • Algorithm
@zhangjinpeng87 zhangjinpeng87 changed the title Feature request: level compaction skip some sst to reduce write amplification PCP-20: level compaction skip some sst to reduce write amplification Nov 6, 2019
@zhangjinpeng87 zhangjinpeng87 changed the title PCP-20: level compaction skip some sst to reduce write amplification PCP-27: level compaction skip some sst to reduce write amplification Nov 6, 2019
@zhangjinpeng87
Copy link
Member Author

@huachaohuang Seems you have finish this issue facebook/rocksdb#6021, Do you have interest to participate our performance challenge match? https://pingcap.com/community-cn/tidb-performance-challenge/

@huachaohuang
Copy link

Well, I'm not sure, I just do that for fun. Thanks for letting me know.

@huachaohuang
Copy link

How's the progress of this issue, are you still working on facebook/rocksdb#5201?
I'm interested on using this feature now.

@yiwu-arbug
Copy link
Collaborator

How's the progress of this issue, are you still working on facebook/rocksdb#5201?
I'm interested on using this feature now.

I'm restarting the work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants