Add capability of prefetch for rafs v6 format #319
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bergwolf
reviewed
Mar 4, 2022
| bios.iter_mut().for_each(|b| { | ||
| if let Some(ref chunks_meta) = self.meta { | ||
| // TODO: the first blob backend io triggers chunks array download. | ||
| if let BlobIoChunk::Address(_blob_index, chunk_index) = b.chunkinfo { |
There was a problem hiding this comment.
Not related to this PR, but it looks a bit wired to construct BlobIoChunk::Address and convert it to BlobIoChunk::Base for v6 IO desc. Looks like we can just create BlobIoChunk::Base and remove BlobIoChunk::Address. @jiangliu wdyt?
hsiangkao
reviewed
Mar 9, 2022
rafs/src/metadata/layout/v6.rs
Outdated
| pub struct RafsV6PrefetchTable { | ||
| /// List of inode numbers for prefetch. | ||
| /// Note: It's not inode index of inodes table being stored here. | ||
| pub inodes: Vec<u64>, |
There was a problem hiding this comment.
Hi, we could use 32bit nid here, since blob.meta won't be too large for nydus v6 format.
it can support 4*32 = 128GB bootstrap size, which I think it's enough
There was a problem hiding this comment.
already changed to u32
imeoer
approved these changes
Mar 9, 2022
jiangliu
approved these changes
Mar 9, 2022
Place a prefetch table which has nids for files and directories that needs to be prefetched. The prefetch table is very close to blobs table on v6 rafs extended super block. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Load v6 prefetch table from extended super block and prefetch files' contents per as the table. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Because seek is transitive. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Rafs prefetch needs the functionality. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
V6 format does not have chunks' particualer addresses in bootstrap but only their indexes by which to locate each chunk from blob. For prefetch, we aslo have to map the chunks info so that bios can be sorted when handling prefetch requests. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Right now, when prefetch enabled and xattrs packed, extended sb will be revised and rewritten more than onece. It is not necessary. Only write extended block when all the settings are done. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rafs v6 now has the capability of prefetch with this patch set merged if it would.
By design, it places a prefetch table very closely to blob devices table on v6 extended block when creating v6 nydus image.
The
nydus-imagecommand line user interface is exactly the same with v5 style. Each file that needs to be prefetched will have itsnidpersisted into the prefetch table.When nydusd starts, the prefetching files will be found from the prefetch table and the files' contents will be downloaded in ahead of time.