diff --git a/README.md b/README.md index b57bd65d8cc..332e7da66af 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ The following Benchmarking results demonstrate that Nydus images significantly o | Storage | Registry/OSS/S3/NAS | Support for OCI-compatible distribution implementations such as Docker Hub, Harbor, Github GHCR, Aliyun ACR, NAS, and Aliyun OSS-like object storage service | ✅ | | Storage/Build | [Harbor](https://github.com/goharbor/acceleration-service) | Provides a general service for Harbor to support acceleration image conversion based on kinds of accelerator like Nydus and eStargz etc | ✅ | | Distribution | [Dragonfly](https://github.com/dragonflyoss/Dragonfly2) | Improve the runtime performance of Nydus image even further with the Dragonfly P2P data distribution system | ✅ | -| Build | [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md) | Provides the ability to build and export Nydus images directly from Dockerfile | ✅ | +| Build | [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md) | Provides the ability to build and export Nydus images directly from Dockerfile | ✅ | | Build/Runtime | [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md) | The containerd client to build or run (requires nydus snapshotter) Nydus image | ✅ | | Runtime | [Docker / Moby](https://github.com/dragonflyoss/nydus/blob/master/docs/docker-env-setup.md) | Run Nydus image in Docker container with containerd and nydus-snapshotter | ✅ | | Runtime | [Kubernetes](https://github.com/containerd/nydus-snapshotter/blob/main/docs/run_nydus_in_kubernetes.md) | Run Nydus image using CRI interface | ✅ | @@ -89,7 +89,7 @@ make docker-static Convert OCIv1 image to Nydus image: [Nydusify](./docs/nydusify.md), [Acceld](https://github.com/goharbor/acceleration-service) or [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md#build-nydus-image-using-nerdctl-image-convert). -Build Nydus image from Dockerfile directly: [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md). +Build Nydus image from Dockerfile directly: [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md). Build Nydus layer from various sources: [Nydus Image Builder](./docs/nydus-image.md). diff --git a/docs/containerd-env-setup.md b/docs/containerd-env-setup.md index fd610be593d..e4f783e793c 100644 --- a/docs/containerd-env-setup.md +++ b/docs/containerd-env-setup.md @@ -16,6 +16,26 @@ sudo install -D -m 755 nydusd nydus-image nydusify nydusctl nydus-overlayfs /usr sudo install -D -m 755 containerd-nydus-grpc /usr/bin ``` +## Start a Local Registry Container + +To make it easier to convert and run nydus images next, we can run a local registry service with docker: + +```bash +sudo docker run -d --restart=always -p 5000:5000 registry +``` + +## Convert/Build an Image to Nydus Format + +Nydus image can be created by converting from an existing OCI or docker v2 image stored in container registry or directly built from Dockerfile(with [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md)) + +Note: For private registry repo, please make sure you are authorized to pull and push the target registry. The basic method is to use `docker pull` and `docker push` to verify your access to the source or target registry. + +```bash +sudo nydusify convert --source ubuntu --target localhost:5000/ubuntu-nydus +``` + +For more details about how to build nydus image, please refer to [Nydusify](https://github.com/dragonflyoss/nydus/blob/master/docs/nydusify.md) conversion tool, [Acceld](https://github.com/goharbor/acceleration-service) conversion service or [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md#build-nydus-image-using-nerdctl-image-convert). + ## Start Nydus Snapshotter Nydus provides a containerd remote snapshotter `containerd-nydus-grpc` (nydus snapshotter) to prepare container rootfs with nydus formatted images. @@ -82,7 +102,7 @@ sudo /usr/bin/containerd-nydus-grpc \ --log-to-stdout ``` -## Configure and Start Containerd +## [Option 1] Configure as Containerd Global Snapshotter Nydus depends on two features of Containerd: @@ -107,6 +127,7 @@ For version 1 containerd config format: [plugins.cri.containerd] snapshotter = "nydus" disable_snapshot_annotations = false + discard_unpacked_layers = false ``` For version 2 containerd config format: @@ -115,6 +136,7 @@ For version 2 containerd config format: [plugins."io.containerd.grpc.v1.cri".containerd] snapshotter = "nydus" disable_snapshot_annotations = false + discard_unpacked_layers = false ``` Then restart containerd, e.g.: @@ -123,19 +145,45 @@ Then restart containerd, e.g.: sudo systemctl restart containerd ``` -### Using Runtime-level Snapshotter in Container for Nydus -Containerd (>= v1.7.0) supports configuring the ```runtime-level``` snapshotter. By following the steps below, we can declare runtimes that use different snapshotters: -#### Step 1: Configure Containerd +## [Option 2] Configure as Containerd Runtime-Level Snapshotter + +Note: this way only works on CRI based scenario (for example crictl or kubernetes). + +Containerd (>= v1.7.0) supports configuring the `runtime-level` snapshotter. By following the steps below, we can declare runtimes that use different snapshotters: + +### Step 1: Apply Containerd Patches + +[Patch](https://github.com/nydusaccelerator/containerd/commit/0959cdb0b190e35c058a0e5bc2e256e59b95b584): fixes the handle of sandbox run and container create for runtime-level snapshotter; + +### Step 2: Configure Containerd + +Only for version 2 containerd config format: + ```toml [plugins."io.containerd.grpc.v1.cri".containerd] snapshotter = "overlayfs" disable_snapshot_annotations = false discard_unpacked_layers = false - [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nydus-runc] + + [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc-nydus] snapshotter = "nydus" + +[proxy_plugins] + [proxy_plugins.nydus] + type = "snapshot" + address = "/run/containerd-nydus/containerd-nydus-grpc.sock" +``` + +Then restart containerd, e.g.: + +```bash +sudo systemctl restart containerd ``` -#### Step 2: Add an Extra Annotation in Sandbox Spec -The annotation ```"io.containerd.cri.runtime-handler": "nydus-runc"``` must be set in sandbox spec. The ```nydus-sandbox.yaml``` looks like below: + +### Step 3: Add an Extra Annotation in Sandbox Spec + +The annotation `"io.containerd.cri.runtime-handler": "runc-nydus"` must be set in sandbox spec. The `nydus-sandbox.yaml` looks like below: + ```yaml metadata: attempt: 1 @@ -147,29 +195,24 @@ linux: namespace_options: network: 2 annotations: - "io.containerd.cri.runtime-handler": "nydus-runc" + "io.containerd.cri.runtime-handler": "runc-nydus" ``` -As shown above, the sandbox is declared with ```"io.containerd.cri.runtime-handler": "nydus-runc"``` annotation will use the ```nydus``` snapshotter, while others will use the default ```overlayfs``` snapshotter. -## Start a Local Registry Container +As shown above, the sandbox is declared with `"io.containerd.cri.runtime-handler": "runc-nydus"` annotation will use the `nydus` snapshotter, while others will use the default `overlayfs` snapshotter. -To make it easier to convert and run nydus images next, we can run a local registry service with docker: +Note: You may encounter the following error when creating a Pod: -```bash -sudo docker run -d --restart=always -p 5000:5000 registry +``` +err="failed to \"StartContainer\" for \"xxx\" with CreateContainerError: \"failed to create containerd container: error unpacking image: failed to extract layer sha256:yyy: failed to get reader from content store: content digest sha256:zzz: not found\"" ``` -## Convert/Build an Image to Nydus Format - -Nydus image can be created by converting from an existing OCI or docker v2 image stored in container registry or directly built from Dockerfile(with [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md)) +This is because some images in the Pod (including the Pause image) have used containerd's default snapshotter (such as the `overlayfs`` snapshotter), and the `discard_unpacked_layers` option was previously set to `true`, containerd has already deleted the blobs from the content store. To resolve this issue, you should first ensure that `discard_unpacked_layers=false`, then use the following command to restore the image: -Note: For private registry repo, please make sure you are authorized to pull and push the target registry. The basic method is to use `docker pull` and `docker push` to verify your access to the source or target registry. - -```bash -sudo nydusify convert --source ubuntu --target localhost:5000/ubuntu-nydus +``` +ctr -n k8s.io content fetch pause:3.8 ``` -For more details about how to build nydus image, please refer to [Nydusify](https://github.com/dragonflyoss/nydus/blob/master/docs/nydusify.md) conversion tool, [Acceld](https://github.com/goharbor/acceleration-service) conversion service or [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md#build-nydus-image-using-nerdctl-image-convert). +Please note that `pause:3.8` is just an example image, you should also fetch all images used by the Pod to ensure that there are no issues. ## Try Nydus with `nerdctl` @@ -195,8 +238,6 @@ linux: security_context: namespace_options: network: 2 -annotations: - "io.containerd.osfeature": "nydus.remoteimage.v1" ``` The `nydus-container.yaml` looks like below: diff --git a/docs/nydusd.md b/docs/nydusd.md index 6697bb326f0..a5ecc34d659 100644 --- a/docs/nydusd.md +++ b/docs/nydusd.md @@ -145,6 +145,8 @@ We are working on enabling cloud-hypervisor support for nydus. #### Use Different Storage Backends +Using different storage backend means that the nydus image metadata (bootstrap) layer is stored in the image registry, but the data layer will be stored on the external storage. Therefore, the option `--target` for `nydusify convert` is still required, the registry image reference is needed to store the metadata layer. + ##### Localfs Backend ``` @@ -192,7 +194,7 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/ } ``` -##### OSS backend with blobcache +##### OSS Backend ``` { @@ -204,7 +206,32 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/ "endpoint": "region.aliyuncs.com", "access_key_id": "", "access_key_secret": "", - "bucket_name": "" + "bucket_name": "", + "object_prefix": "nydus/" + } + }, + ... + }, + ... +} +``` + +##### S3 Backend + +``` +{ + "device": { + "backend": { + "type": "s3", + "config": { + ... + "endpoint": "s3.amazonaws.com", + "scheme": "https", + "access_key_id": "", + "access_key_secret": "", + "bucket_name": "", + "region": "", + "object_prefix": "nydus/" } }, ... @@ -213,7 +240,7 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/ } ``` -##### Registry backend +##### Registry Backend ``` { @@ -246,41 +273,51 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/ ``` Note: The value of `device.backend.config.auth` will be overwrite if running the nydusd with environment variable `IMAGE_PULL_AUTH`. -##### Enable P2P Proxy for Storage Backend +#### HTTP Proxy Backend -Add `device.backend.config.proxy` field to enable HTTP proxy for storage backend. For example, use P2P distribution service to reduce network workload and latency in large scale container cluster using [Dragonfly](https://d7y.io/) (enable centralized dfdaemon mode). +The `HttpProxy` backend can access blobs through a http proxy server which can be local (using unix socket) or remote (using `https://` or using `http://`). + +`HttpProxy` uses two API endpoints to access the blobs: +- `HEAD /path/to/blobs` to get the blob size +- `GET /path/to/blobs` to read the blob + +The http proxy server should respect [the `Range` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-range) to compute the offset and length of the blob. + +The example config files for the `HttpProxy` backend may be: ``` +// for remote usage { "device": { - "backend": { - "type": "registry", + "backend": { + "type": "http-proxy", "config": { - "proxy": { - // Access remote storage backend via P2P proxy, e.g. Dragonfly dfdaemon server URL - "url": "http://p2p-proxy:65001", - // Fallback to remote storage backend if P2P proxy ping failed - "fallback": true, - // Endpoint of P2P proxy health checking - "ping_url": "http://p2p-proxy:40901/server/ping", - // Interval of P2P proxy health checking, in seconds - "check_interval": 5 - }, - ... + "addr": "http://127.0.0.1:9977", + "path": "/namespace//blobs" } - }, - ... - }, - ... + } + } } ``` -Once the configuration is loaded successfully on nydusd starting, we will see the log as shown below: +or ``` -INFO [storage/src/backend/connection.rs:136] backend config: CommonConfig { proxy: ProxyConfig { url: "http://p2p-proxy:65001", ping_url: "http://p2p-proxy:40901/server/ping", fallback: true, check_interval: 5 }, timeout: 5, connect_timeout: 5, retry_limit: 0 } +// for local usage +{ + "device": { + "backend": { + "type": "http-proxy", + "config": { + "addr": "/path/to/unix.sock", + } + } + } +} ``` +The `HttpProxy` backend also supports the `Proxy` and `Mirrors` configurations for remote usage like the `Registry backend` described above. + ##### Enable Mirrors for Storage Backend (Recommend) Nydus is deeply integrated with [Dragonfly](https://d7y.io/) P2P mirror mode, please refer the [doc](https://d7y.io/docs/setup/integration/nydus) to learn how configuring Nydus to use Dragonfly. @@ -332,51 +369,42 @@ Currently, the mirror mode is only tested in the registry backend, and in theory } ``` -#### HTTP proxy backend - -The `HttpProxy` backend can access blobs through a http proxy server which can be local (using unix socket) or remote (using `https://` or using `http://`). - -`HttpProxy` uses two API endpoints to access the blobs: -- `HEAD /path/to/blobs` to get the blob size -- `GET /path/to/blobs` to read the blob -The http proxy server should respect [the `Range` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-range) to compute the offset and length of the blob. +##### Enable P2P Proxy for Storage Backend -The example config files for the `HttpProxy` backend may be: +Add `device.backend.config.proxy` field to enable HTTP proxy for storage backend. For example, use P2P distribution service to reduce network workload and latency in large scale container cluster using [Dragonfly](https://d7y.io/) (enable centralized dfdaemon mode). ``` -// for remote usage { "device": { - "backend": { - "type": "http-proxy", + "backend": { + "type": "registry", "config": { - "addr": "http://127.0.0.1:9977", - "path": "/namespace//blobs" + "proxy": { + // Access remote storage backend via P2P proxy, e.g. Dragonfly dfdaemon server URL + "url": "http://p2p-proxy:65001", + // Fallback to remote storage backend if P2P proxy ping failed + "fallback": true, + // Endpoint of P2P proxy health checking + "ping_url": "http://p2p-proxy:40901/server/ping", + // Interval of P2P proxy health checking, in seconds + "check_interval": 5 + }, + ... } - } - } + }, + ... + }, + ... } ``` -or +Once the configuration is loaded successfully on nydusd starting, we will see the log as shown below: ``` -// for local usage -{ - "device": { - "backend": { - "type": "http-proxy", - "config": { - "addr": "/path/to/unix.sock", - } - } - } -} +INFO [storage/src/backend/connection.rs:136] backend config: CommonConfig { proxy: ProxyConfig { url: "http://p2p-proxy:65001", ping_url: "http://p2p-proxy:40901/server/ping", fallback: true, check_interval: 5 }, timeout: 5, connect_timeout: 5, retry_limit: 0 } ``` -The `HttpProxy` backend also supports the `Proxy` and `Mirrors` configurations for remote usage like the `Registry backend` described above. - ### Mount Bootstrap Via API To mount a bootstrap via api, first launch nydusd without a bootstrap: