nydusify & nydus-image: v6 image conversion support #473
Conversation
…mlink file Only the dir and symlink file of rafsv6 can be of EROFS_INODE_FLAT_INLINE layout, so we only need to compute dirents offset for these two type of files. Otherwise, it may overflow. Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com> Signed-off-by: Qi Wang <wangqi@linux.alibaba.com>
…uild First, we need to change v5 specific interface to universal interface. Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com> Signed-off-by: Qi Wang <wangqi@linux.alibaba.com>
We implement unimplmented methods of v6's OndiskInodeWrapper. Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com> Signed-off-by: Qi Wang <wangqi@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com> Signed-off-by: Qi Wang <wangqi@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Previously, in some methods, we assume the `.` and `..` are the first two dirents, which is not true, both of them are globally sorted along with other dirents, we fix that in this patch. Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
When build centos and java's v6 image, the conversion failed due to incorrent xattr index. We fix that in this patch. Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Previously, we have modified the nydus-image to allow specifying parent bootstrap when converting v6 image, therefore, we add the image-version argument into nydusify to support v6 image conversion. Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
…h nydus-image Signed-off-by: Qi Wang <wangqi@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
|
@liubogithub @jiangliu @imeoer PTAL |
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
| let mut bootstrap = Bootstrap::new()?; | ||
| if bootstrap_ctx.layered { | ||
| // Merge with lower layer if there's one, do not prepare `prefetch` list during merging. | ||
| ctx.prefetch.disable(); | ||
| bootstrap.build(ctx, &mut bootstrap_ctx, &mut tree)?; | ||
| tree = bootstrap.apply(ctx, &mut bootstrap_ctx, bootstrap_mgr, blob_mgr, None)?; | ||
| } | ||
| // If layered, the bootstrap_ctx.offset will be set in first build, so we need restore it here | ||
| bootstrap_ctx.offset = origin_bootstarp_offset; | ||
| bootstrap_ctx.layered = false; |
There was a problem hiding this comment.
This also affects https://github.com/dragonflyoss/image-service/blob/fdee5992103916910dea755a2df3d5d669ea6efb/src/bin/nydus-image/core/bootstrap.rs#L202
where layered is used to check whiteout, have you checked if setting layered = false would cause any difference?
There was a problem hiding this comment.
In fact, the processing of whiteout is already done at the time of the first bootstarp.build, and here is actually the second time to do bootstarp.build
if bootstrap_ctx.layered {
// Merge with lower layer if there's one, do not prepare `prefetch` list during merging.
ctx.prefetch.disable();
bootstrap.build(ctx, &mut bootstrap_ctx, &mut tree)?;
tree = bootstrap.apply(ctx, &mut bootstrap_ctx, bootstrap_mgr, blob_mgr, None)?;
}
BTW, https://github.com/dragonflyoss/image-service/blob/master/contrib/nydusify/tests/texture/image-from-2/Dockerfile#L12. There are some integration tests for whiteout in nydusify, which are currently passed.
| for j in 0..entries_count { | ||
| let de = self | ||
| .get_entry(i as usize, j as usize) | ||
| .map_err(err_invalidate_data)?; |
There was a problem hiding this comment.
err_invalidate_data is confusing, would you please change it to err_inval_data in a separate patch?
| @@ -109,7 +109,7 @@ fn test( | |||
| cache_compressed, | |||
| rafs_mode.parse().unwrap(), | |||
| "api.sock".into(), | |||
| true, | |||
| !rafsv6, | |||
There was a problem hiding this comment.
Can you please leave a comment either here or in the patch log for why digest validate for v6 is disabled.
There was a problem hiding this comment.
Actually already has some comments. exists only where it was first used.
// FIXME: Currently no digest validation is implemented for rafs v6.
!rafsv6,
| @@ -141,20 +181,19 @@ impl RafsInspector { | |||
|
|
|||
| fn load_meta(f: &mut RafsIoReader) -> Result<(RafsMeta, RafsLayout)> { | |||
| let layout_profile = RafsLayout::rafsv5_layout(); | |||
| match Self::super_block_v5(f, &layout_profile) { | |||
| Ok(sb) => { | |||
| let mut res = Self::super_block_v5(f, &layout_profile).map(|sb| { | |||
There was a problem hiding this comment.
super_block_v5 does RafsV5SuperBlock::load() which doesn't check fs magic or fs version, so it won't return errors if it's a v6 superblock.
There was a problem hiding this comment.
I modified the super_block_v5 and added a checker for rafs versions.
if !sb.detect() {
return Err(anyhow!("invalid rafs v5 version"));
}
|
@liubogithub PTAL |
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
Signed-off-by: zyfjeff <zyfjeff@linux.alibaba.com>
We have adapted the nydusify and nydus-image to support using nydusify directly to build v6 image, to accomplish this purpose, we have introduce several features and fix several bugs.
Features:
Bug fixes:
FYI, we have tested converting all top images in the image_list.txt and checked their bootstraps with fsck.erofs.
#425