Introduce nydus-v2.0/rafsv6 #234
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Simpilify refine Cargo.toml for readability. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Refine `trait BlobBackend` by: - introducing `trait BlobReader` and `trait BlobWriter` - `BlobBackend` now is a manager to manage reader and writer - `BlobBackend` will be extend to support GC later Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Refine LocalFs storage backend by: - migrate to the new Backend interfaces - redesign the access trace and prefetching logic - add unit test cases Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Refine request.rs by: - add some documentation - rename build_client() as build_connection() - add Request::shutdown() to teardown background working threads - add some unit test cases Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Refine oss storage backend by: - migrate to the new backend interfaces - add more unit tests Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Refine registry storage backend by: - migrate to the new backend interfaces - add more unit tests Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
The request actually stands for connection, so rename it. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
enhance connection::respond() to simplify code. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Rename Backend::release() to Backend::shutdown(), and explicitly shutdown all background working threads. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
enhance copyv() and MemSliceCrusor in util by: - stricter validation of input parameter - simplify code - add more unit test cases - fix a bug in BlobReader::readv Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Refine RafsCache related code and documentation by: - adding more documentation - introduce function compress::compute_compressed_gzip_size() - validate data size returned by backend - rename BlobCache::release() as BlobCache::destroy() Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Explicitly group Rafs v5 specific logic by: - move Rafs v5 specific data structures into submodule v5 - also rename device::RafsXXX to device::BlobXXX. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Refine ChunkMap implementations by: - get rid of dependency on v5 data struct by using BlobChunkInfo instead of BlobV5ChunkInfo - export DigestedChunkMap and IndexedChunkMap directly - rename ChunkMap::has_ready() as ChunkMap::is_ready() - rename ChunkMap::has_ready_nowait() as ChunkMap::is_ready_nowait() - rename ChunkMap::finish() as ChunkMap::notify_ready() - Make Status::wait_for_inflight() return current value Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
There's possible race condition in IndexedChunkMap::new(), so refine it to close those race windows. Introduce IndexedChunkMap v1, which has a flag for "all chunk ready". Also add more unit test cases. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Introduce NoopChunkMap, which will simplify report all chunk as ready. The NoopChunkMap may be used support disk based blob storage. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Improve documentation for chunkmap. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Move RafsCache into cache::v5::BlobV5Cache, so all rafs V5 related code into dedicated submodule. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Introduce trait BlobCacheMgr/BlobCache and migrate dummycache.rs to the new interfaces. blobcache.rs will handled with dedicated changes. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Introduce `BlobIoChunk` enumeration and refactor `BlobV5Io/BlobV5IoDesc` as `BlobIoDesc/BlobIoVec`. Now BlobIoChunk/BlobIoDesc/BlobIoVec becomes rafs version independent, which is essential to enhance blobcache.rs to support rafs v6. With this refactor, the abstraction becomes much more elegant. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Enhance the DummyCacheMgr by: - explicitly control prefetching behavior - more docs - optimize DummyCacheMgr::read() for rafs v5 Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Enhance BlobInfo to provide more information about compression and digest algorithm etc, then we could make these configuration as per blob instead of per cache manager. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Make all fields of BlobInfo as private and provide access methods for them. Also rename BlobChunkInfo::block_id() as BlobChunkInfo::chunk_id(), as there's no any annotation of 'block' within the storage subsystem. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Temporarily disable blobcache, will be added back later. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Refactor interfaces of the cache subsystem to move main operation from BlobCacheMgr to BlobCache. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
- rename MergedBackendRequest to ChunkIoMerged - rename ChunkIoInfo to ChunkIoTag - with help of BlobIoChunk, make ChunkIoMerged independent of v5 - enhance state validation in MergedBackendRequest::new/merge() Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Add read(), read_chunks(), read_raw_chunk() and process_raw_chunk() to BlobCache, so a BlobCache object could be used to access blob data. Rename BlobCache::is_chunk_cached() to is_chunk_ready(). Also optimize copyv() to avoid create a Vec<&[u8]> from Vec<Vec<u8>>. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
BlobV5Cache actually represents a blob cache object instead of a blob cache manager object, so change pub trait BlobV5Cache: BlobCacheMgr" to "pub trait BlobV5Cache: BlobCache". With this refactor, we could remove much v5 specific code. Also refactor the factory module to explicity manage blob cache manager and provide interface to support gc. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
Now cache::v5::BlobV5Cache is not used anymore, so remove it. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
One big difference is that v6 blob may use different compressor and digester with image's default ones. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com> Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Special inode like device, char, pipe are now supported. Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Given OCI image spec supports sock file, lets stick to it. Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Although we still use v5 inode/chunk, we can use v6 wrapper for better naming. Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Set rafs v6's blocks of superblock to a non-zero value. Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
This adds a few verbose errors for validate(). Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Now that we've put a sanity check in RafsV6Blob's validate(), this makes blob's uncompressed size align to 4096 which is EROFS_BLOCK_SIZE in use. Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
UT needs to be updated after Node honors "version". Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
In order to exit more gracefully. Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
size_with_xattr() is only for v6, lets put a dummy value for v5. Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Even we exclude this option from nydusd with feature fusedev applied, it can still be used from API mount. Like libfuse, we just want to provide a simple example that built-in fuse-rs works well and test can be composed to verify code in nydusd/rafs when linking to virtiofs front-end. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Async prefetch is always started when nydusd starts. Files that have to be prefetched in a list will be sent to filecache async workers manager. After all file hint are sent via channel, we should call `stop_prefetch` to release resources synchronously, such as worker threads. Signed-off-by: Changwei Ge <chge@linux.alibaba.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Temporary workadround. Otherwise, the rest prefetch requests will be dropped directly. Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Some functions will be used in the near future, so `allow_dead` code to silent compiling warnings temporarily. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Builder panic on multi-layers build: ``` thread 'main' panicked at 'range end index 4293918793 out of range for slice of length 1048576', src/bin/nydus-image/core/node.rs:307:39` ``` The `dir_set_v6_offset` method will rewrite the inode size of regular file, that causes the above issue. Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
If blob's ci compressor is None, the uncompressed buffer should be read out and returned. Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Thus to make it more suggestive Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
It is not always rafs. Signed-off-by: Peng Tao <bergwolf@hyper.sh>
For rafs, we would like to disable open. For passthroughfs, we would like to enable open and killpriv_v2. Signed-off-by: Peng Tao <bergwolf@hyper.sh>
To ask kernel to cache file and directory contents instead of dropping them across open/close. Signed-off-by: Peng Tao <bergwolf@hyper.sh>
So that we can enable necessary options for passthroughfs when running rafs and passthroughfs under the same vfs. Signed-off-by: Peng Tao <bergwolf@hyper.sh>
Current inode loading logic wrongly used inode number to load inode from inodes table. Signed-off-by: Changwei Ge <chge@linux.alibaba.com>
Clean clippy warning by `cargo fmt`. Signed-off-by: Liu Jiang <gerry@linux.alibaba.com>
liubogithub
approved these changes
Dec 3, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patchset is a little big(huge), which does: