Adopt zlib random access algorithm to support OCIv1 compatible nydus image #833
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thread 'main' panicked at 'Command create: Argument or group 'estargztoc-ref' specified in 'requires*' for 'blob-id' does not exist', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-4.0.18/src/builder/debug_asserts.rs:152:13 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Change storage interfaces to relax the constraint that compressed chunk must be continuous. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Refine implementation of legacy stargz. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Simplify RAFS fs prefetch implementation. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Refine data validation related logic: 1) force validating of data fetched from storage backend 2) disable data validation for legacy stargz images. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Amplify read requests for RAFS v6 at storage layer, this may help to reduce IO requests to storage backend and improve performance. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Fix a bug in generating chunk info for RAFS Zran fs. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Enhance nydusd to support ZRan based RAFS v6 fs. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Support generating ZRan based RAFS fs from estargz file, which is similiar to 'targz-to-ref' but filtering out all estargz specific files. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
Only support generating RAFS v6 from estargz/targz files. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
|
@jiangliu , a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/30836 |
|
@jiangliu , The CI test is completed, please check result:
Congratulations, your test job passed! |
||||||||||||||||||||||||
|
Fixes #686 |
imeoer
approved these changes
Nov 7, 2022
|
Let's move forward quickly. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By adopting zlib random access algorithm, nydus can build a RAFS metadata from a targz file and use the targz file as RAFS data blob. By this way, we could associate a small RAFS metadata blob to existing OCIv1 images, which provides lazy loading and data deduplication capabilities with minimal resource overhead.
Now we have a flexible architecture to support all of OCIv1/eStargz/Nydus images as below: