Baseline Resync Enhancement #244
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yuwmao
force-pushed
the
baseline_resync
branch
from
b831af9 to
2a7e47a
Compare
yuwmao
commented
Dec 23, 2024
| @@ -379,17 +379,6 @@ void HSHomeObject::on_shard_message_commit(int64_t lsn, sisl::blob const& h, hom | |||
| local_create_shard(shard_info, v_chunk_id, blkids.chunk_num(), blkids.blk_count()); | |||
| if (ctx) { ctx->promise_.setValue(ShardManager::Result< ShardInfo >(shard_info)); } | |||
|
|
|||
| // update pg's total_occupied_blk_count | |||
There was a problem hiding this comment.
Redundant, these codes are already put into the local_create_shard.
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## baseline_resync #244 +/- ##
===================================================
- Coverage 65.52% 64.25% -1.27%
===================================================
Files 33 33
Lines 2271 2417 +146
Branches 254 281 +27
===================================================
+ Hits 1488 1553 +65
- Misses 659 719 +60
- Partials 124 145 +21 ☔ View full report in Codecov by Sentry. |
yuwmao
force-pushed
the
baseline_resync
branch
2 times, most recently
from
556376d to
f743e81
Compare
yuwmao
changed the title
yuwmao
force-pushed
the
baseline_resync
branch
2 times, most recently
from
ce6769f to
b07084c
Compare
yuwmao
force-pushed
the
baseline_resync
branch
from
b07084c to
9ef9f5f
Compare
yuwmao
changed the title
This commit addresses blob duplication issues caused by resync, it mainly happens when resend snapshot mesg during baseline resync and apply log after snapshot completion. This helps avoid unnecessary GC due to duplicated data. This mechanism is effective only for duplicated blobs. Since we do not record the blks of the `shardInfo` stored in the data service, we are unable to skip data writes for duplicated shards.
yuwmao
force-pushed
the
baseline_resync
branch
5 times, most recently
from
39b167b to
9602e86
Compare
xiaoxichen
reviewed
Dec 26, 2024
yuwmao
force-pushed
the
baseline_resync
branch
from
9602e86 to
795b7ff
Compare
xiaoxichen
approved these changes
Dec 26, 2024
koujl
approved these changes
Dec 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR enhances baseline resync:
This mechanism is effective only for duplicated blobs. Since we do not record the blks of the
shardInfostored in the data service, we are unable to skip data writes for duplicated shards.