Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce size of EntryIndex #206

Merged
merged 8 commits into from
Apr 6, 2022
Merged

Reduce size of EntryIndex #206

merged 8 commits into from
Apr 6, 2022

Conversation

tabokie
Copy link
Member

@tabokie tabokie commented Mar 31, 2022

Ref #205

Some enhancements:

  • Do not store log index inside EntryIndex.
  • Reduce entry offset and entry len from u64 to u32. It won't cause regression because the length is already limited by compression (i32::MAX).

Size for one EntryIndex is reduced from 64 bytes to 48 bytes (25%).

stress benchmarks shows 18.7% improvement (compared to results in #207, normalized with write bandwidth):

Memory: 7003MiB
[write]
Throughput(QPS) = 32322.92
Latency(μs) min = 8, avg = 16.10, p50 = 13, p90 = 26, p95 = 31, p99 = 45, p99.9 = 79, max = 4867
Fairness = 100.0%
Write Bandwidth = 24.2MiB/s

tabokie added 2 commits March 31, 2022 20:09
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Copy link
Member

@Connor1996 Connor1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you describe what you change to reduce the size

src/memtable.rs Outdated
assert_eq!(idx.index, last_index + 1);
}
last_index = Some(idx.index);
// if let Some(last_index) = last_index {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why comment out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for this check because entry index doesn't store log index anymore.

@@ -36,9 +36,9 @@ pub struct EntryIndex {
pub compression_type: CompressionType,

/// The relative offset within its group of entries.
pub entry_offset: u64,
pub entry_offset: u32,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrap ThinEntryIndex directly?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No because ThinEntryIndex should be a private struct internal to MemTable.

tabokie added 2 commits April 1, 2022 13:15
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
@codecov
Copy link

codecov bot commented Apr 1, 2022

Codecov Report

Merging #206 (cf520a6) into master (2d1f8fc) will decrease coverage by 0.03%.
The diff coverage is 97.10%.

@@            Coverage Diff             @@
##           master     #206      +/-   ##
==========================================
- Coverage   96.98%   96.95%   -0.04%     
==========================================
  Files          28       28              
  Lines        7602     7652      +50     
==========================================
+ Hits         7373     7419      +46     
- Misses        229      233       +4     
Impacted Files Coverage Δ
src/errors.rs 100.00% <ø> (ø)
src/pipe_log.rs 94.11% <ø> (ø)
src/log_batch.rs 97.29% <93.75%> (-0.38%) ⬇️
src/memtable.rs 99.13% <98.43%> (+0.12%) ⬆️
src/engine.rs 97.17% <100.00%> (+<0.01%) ⬆️
src/file_pipe_log/log_file.rs 100.00% <100.00%> (ø)
src/filter.rs 82.50% <100.00%> (ø)
tests/failpoints/mod.rs 100.00% <100.00%> (ø)
src/consistency.rs 91.11% <0.00%> (-2.23%) ⬇️
src/codec.rs 98.34% <0.00%> (-0.34%) ⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d1f8fc...cf520a6. Read the comment docs.

tabokie added 2 commits April 1, 2022 13:48
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie added 2 commits April 6, 2022 13:14
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
@tabokie tabokie merged commit bac5ec4 into tikv:master Apr 6, 2022
@tabokie tabokie deleted the reduce-memory branch April 6, 2022 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants