Skip to content

Commit

Permalink
Merge pull request #332 from liubin/liubin/dev
Browse files Browse the repository at this point in the history
Some minor fixes
  • Loading branch information
bergwolf authored Mar 16, 2022
2 parents 730b5ea + 3f3b5ff commit 4631c5b
Show file tree
Hide file tree
Showing 7 changed files with 30 additions and 29 deletions.
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ fusedev:
$(call static_check,$@,target-$@)

clean:
cargo clean
cargo clean --target-dir target-virtiofs
cargo clean --target-dir target-fusedev

Expand Down
35 changes: 21 additions & 14 deletions docs/nydus-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Rafs presents to users a userspace filesystem with separating filesystem's metad

* bootstrap
* The metadata is a merkle tree whose nodes represents a regular filesystem's directory/file
* a leaf nodes refers to a file and contains hash value of its file data
* a leaf node refers to a file and contains hash value of its file data
* Root node and internal nodes refer to directories and contain the hash value of their children nodes.
```
+-----------------+-----------------+--------------+----------------+-------+-------+--------+---------------------+
Expand Down Expand Up @@ -86,10 +86,13 @@ As a lazily fetch solution, prefetch plays an important role to mitigate the imp

The image build tool `nydusify` accepts prefetch hints from `stdin`.
## 5. Blob and Blob Cache

* blob

Blob is the data part of a container image, it consists of files' data. Nydus has splitted a file's data into one or more fixed-length (1MB) chunks.

* blob cache

Nydus can be configured to set up a cache for blob, called `blobcache`. With `blobcache`, fetched blob data is saved to a `work dir` and won't be repeatedly fetched. Given the assumption that only a small portion of image is fetched, there is no cache eviction for `blobcache`.

## 6. Compression
Expand Down Expand Up @@ -316,11 +319,11 @@ pub struct OndiskBlobTable {
# III. Manifest of Nydus Format Image
Nydus manifest is designed to be fully compatible with OCI image spec and distribution spec by adding an extra manifest file to store the pointers of nydus bootstrap (i.e. metadata) and blobfile (i.e. data).

## 1. Image Index
A typical image index enabling nydus points to two manifest files, one is the traditional OCI v1 image manifest, the other is the nydus manifest that takes advantage of `platform` and puts `os.features: ["nydus.remoteimage.v1"]` field under `platform`.
## 1. Image Index
A typical image index enabling nydus points to two manifest files, one is the traditional OCI v1 image manifest, the other is the nydus manifest that takes advantage of `platform` and puts `os.features: ["nydus.remoteimage.v1"]` field under `platform`.

```json
{
{
"schemaVersion": 2,
"manifests": [
{
Expand All @@ -347,18 +350,22 @@ Nydus manifest is designed to be fully compatible with OCI image spec and distri
]
}
```
## 2. Image Manifest
A typical image manifest of nydus consists of `config.json`, one nydus metadata layer (`"mediaType": "application/vnd.oci.image.layer.v1.tar.gz"`) and one or more nydus data layers (`"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1"`).
* nydus metadata layer
This layer refers to the metadata part of files and directories in the image, including rafs filesystem metadata and digest for validation purpose.
The special part is that an annotation `"containerd.io/snapshot/nydus-bootstrap": "true"` is set and lets containerd's snapshotter know it's nydus metadata layer and do its job accordingly.
* nydus data layer
This layer refers to the data part, please note that the data layers of an image can be owned solely by this image or shared by others, similarly, each data layer is annotated with `"containerd.io/snapshot/nydus-blob": "true"`, which can be used to tell containerd's snapshotter to skip downloading them.

The manifest is designed to be compatible with the dependency architect and garbage collection algorithm widely used by containerd and registry.
## 2. Image Manifest

A typical image manifest of nydus consists of `config.json`, one nydus metadata layer (`"mediaType": "application/vnd.oci.image.layer.v1.tar.gz"`) and one or more nydus data layers (`"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1"`).

* nydus metadata layer

This layer refers to the metadata part of files and directories in the image, including rafs filesystem metadata and digest for validation purpose.
The special part is that an annotation `"containerd.io/snapshot/nydus-bootstrap": "true"` is set and lets containerd's snapshotter know it's nydus metadata layer and do its job accordingly.

* nydus data layer
This layer refers to the data part, please note that the data layers of an image can be owned solely by this image or shared by others, similarly, each data layer is annotated with `"containerd.io/snapshot/nydus-blob": "true"`, which can be used to tell containerd's snapshotter to skip downloading them.

The manifest is designed to be compatible with the dependency architect and garbage collection algorithm widely used by containerd and registry.

```json
{
{
"schemaVersion": 2,
"mediaType": "",
"config": {
Expand Down
2 changes: 1 addition & 1 deletion rafs/src/fs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ fn default_amplify_io() -> u32 {
/// Configuration information for filesystem data prefetch.
#[derive(Clone, Default, Deserialize)]
pub struct FsPrefetchControl {
/// Whether the filesystem layer data prefetch is enable or not.
/// Whether the filesystem layer data prefetch is enabled or not.
#[serde(default)]
pub enable: bool,

Expand Down
2 changes: 1 addition & 1 deletion rafs/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
//! by chunk content digest value. When Rafs file is used for container images, Rafs metadata blob
//! contains all filesystem metadatas, such as directory, file name, permission etc. Actually file
//! contents are divided into chunks and stored into data blobs. Rafs may built one data blob for
//! each container image layer or build a single data blob for the whole image, according to
//! each container image layer or build a single data blob for the whole image, according to
//! building options.
//!
//! There are several versions of Rafs filesystem defined:
Expand Down
1 change: 0 additions & 1 deletion rafs/src/metadata/direct_v6.rs
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ impl DirectMappingState {
base: std::ptr::null(),
end: std::ptr::null(),
size: 0,
// mmapped_inode_table: false,
validate_digest,
}
}
Expand Down
2 changes: 1 addition & 1 deletion rafs/src/metadata/layout/v6.rs
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ impl RafsV6SuperBlock {
self.magic() == EROFS_SUPER_MAGIC_V1
}

/// Get maximum ino.
/// Set number of inodes.
pub fn set_inos(&mut self, inos: u64) {
self.s_inos = inos.to_le();
}
Expand Down
16 changes: 5 additions & 11 deletions rafs/src/metadata/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -453,17 +453,11 @@ impl Default for RafsSuper {
impl RafsSuper {
/// Create a new `RafsSuper` instance from a `RafsConfig` object.
pub fn new(conf: &RafsConfig) -> Result<Self> {
let mut rs = Self::default();

match conf.mode.as_str() {
"direct" => rs.mode = RafsMode::Direct,
"cached" => rs.mode = RafsMode::Cached,
_ => return Err(einval!("Rafs mode should be 'direct' or 'cached'")),
}

rs.validate_digest = conf.digest_validate;

Ok(rs)
Ok(Self {
mode: RafsMode::from_str(&conf.mode.as_str())?,
validate_digest: conf.digest_validate,
..Default::default()
})
}

/// Destroy the filesystem super block.
Expand Down

0 comments on commit 4631c5b

Please sign in to comment.