fix(cli): deduplicate dag stat blocks by multihash #11098
Draft
+40
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
since Kubo v0.12.0, blocks are stored by multihash, so identical data with different CIDs (e.g., CIDv0 vs CIDv1) is stored once. dag stat now reflects actual storage by using multihash-based deduplication instead of CID-based. updated help text to clarify deduplication behavior and note that dag export uses CID-based keying and may include duplicates. added regression test for multihash deduplication.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR closes #8794, but imo is just a formality.
In practice, unlikely this will impact anyone, DAGs with the same block being addressed multiple times with different codec is virtually never happening.
Rationale
since Kubo v0.12.0, blocks are stored by multihash, so identical data with different CIDs (e.g., CIDv0 vs CIDv1) is stored once. dag stat now reflects actual storage by using multihash-based deduplication instead of CID-based.
updated help text to clarify deduplication behavior and note that CAR export (
dag export) uses CID-based keying and may include duplicates. this addresses concern raised in #8843 (comment)added regression test for multihash deduplication.