Skip to content

Commit

Permalink
docs: some improvements for usage
Browse files Browse the repository at this point in the history 
1. buildkit upstream follow-up is slow, update to nydusaccelerator/buildkit;
2. runtime-level snapshotter usage needs some extra containerd patches;
3. add s3 storage backend example for nydusd doc page;

Signed-off-by: Yan Song <yansong.ys@antgroup.com>
  • Loading branch information
@imeoer
imeoer committed Oct 16, 2023
1 parent 5c63ba9 commit b08efb1
Show file tree
Hide file tree
Showing 3 changed files with 105 additions and 64 deletions.
4 changes: 2 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ The following Benchmarking results demonstrate that Nydus images significantly o
| Storage | Registry/OSS/S3/NAS | Support for OCI-compatible distribution implementations such as Docker Hub, Harbor, Github GHCR, Aliyun ACR, NAS, and Aliyun OSS-like object storage service | ✅ |
| Storage/Build | [Harbor](https://github.com/goharbor/acceleration-service) | Provides a general service for Harbor to support acceleration image conversion based on kinds of accelerator like Nydus and eStargz etc | ✅ |
| Distribution | [Dragonfly](https://github.com/dragonflyoss/Dragonfly2) | Improve the runtime performance of Nydus image even further with the Dragonfly P2P data distribution system | ✅ |
| Build | [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md) | Provides the ability to build and export Nydus images directly from Dockerfile | ✅ |
| Build | [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md) | Provides the ability to build and export Nydus images directly from Dockerfile | ✅ |
| Build/Runtime | [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md) | The containerd client to build or run (requires nydus snapshotter) Nydus image | ✅ |
| Runtime | [Docker / Moby](https://github.com/dragonflyoss/image-service/blob/master/docs/docker-env-setup.md) | Run Nydus image in Docker container with containerd and nydus-snapshotter | ✅ |
| Runtime | [Kubernetes](https://github.com/containerd/nydus-snapshotter/blob/main/docs/run_nydus_in_kubernetes.md) | Run Nydus image using CRI interface | ✅ |
Expand All @@ -90,7 +90,7 @@ make docker-static

Convert OCIv1 image to Nydus image: [Nydusify](./docs/nydusify.md), [Acceld](https://github.com/goharbor/acceleration-service) or [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md#build-nydus-image-using-nerdctl-image-convert).

Build Nydus image from Dockerfile directly: [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md).
Build Nydus image from Dockerfile directly: [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md).

Build Nydus layer from various sources: [Nydus Image Builder](./docs/nydus-image.md).

Expand Down
29 changes: 21 additions & 8 deletions docs/containerd-env-setup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ For version 1 containerd config format:
[plugins.cri.containerd]
snapshotter = "nydus"
disable_snapshot_annotations = false
discard_unpacked_layers = false
```

For version 2 containerd config format:
Expand All @@ -115,6 +116,7 @@ For version 2 containerd config format:
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "nydus"
disable_snapshot_annotations = false
discard_unpacked_layers = false
```

Then restart containerd, e.g.:
Expand All @@ -124,18 +126,28 @@ sudo systemctl restart containerd
```

### Using Runtime-level Snapshotter in Container for Nydus
Containerd (>= v1.7.0) supports configuring the ```runtime-level``` snapshotter. By following the steps below, we can declare runtimes that use different snapshotters:
#### Step 1: Configure Containerd

Containerd (>= v1.7.0) supports configuring the `runtime-level` snapshotter. By following the steps below, we can declare runtimes that use different snapshotters:

#### Step 1: Apply Containerd Patches

1. [Patch 1](https://github.com/nydusaccelerator/containerd/commit/0959cdb0b190e35c058a0e5bc2e256e59b95b584): fixes the handle of sandbox run and container create for runtime-level snapshotter;
2. [Patch 2](https://github.com/nydusaccelerator/containerd/commit/b98e71e858a213419c95b79fa0dcfd782ec7a785): always check the content store and fetch the missing blob before the snapshot unpack.

#### Step 2: Configure Containerd

```toml
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "overlayfs"
disable_snapshot_annotations = false
discard_unpacked_layers = false
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nydus-runc]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc-nydus]
snapshotter = "nydus"
```
#### Step 2: Add an Extra Annotation in Sandbox Spec
The annotation ```"io.containerd.cri.runtime-handler": "nydus-runc"``` must be set in sandbox spec. The ```nydus-sandbox.yaml``` looks like below:
#### Step 3: Add an Extra Annotation in Sandbox Spec

The annotation `"io.containerd.cri.runtime-handler": "runc-nydus"` must be set in sandbox spec. The `nydus-sandbox.yaml` looks like below:

```yaml
metadata:
attempt: 1
Expand All @@ -147,9 +159,10 @@ linux:
namespace_options:
network: 2
annotations:
"io.containerd.cri.runtime-handler": "nydus-runc"
"io.containerd.cri.runtime-handler": "runc-nydus"
```
As shown above, the sandbox is declared with ```"io.containerd.cri.runtime-handler": "nydus-runc"``` annotation will use the ```nydus``` snapshotter, while others will use the default ```overlayfs``` snapshotter.
As shown above, the sandbox is declared with `"io.containerd.cri.runtime-handler": "runc-nydus"` annotation will use the `nydus` snapshotter, while others will use the default `overlayfs` snapshotter.

## Start a Local Registry Container

Expand All @@ -161,7 +174,7 @@ sudo docker run -d --restart=always -p 5000:5000 registry

## Convert/Build an Image to Nydus Format

Nydus image can be created by converting from an existing OCI or docker v2 image stored in container registry or directly built from Dockerfile(with [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md))
Nydus image can be created by converting from an existing OCI or docker v2 image stored in container registry or directly built from Dockerfile(with [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md))

Note: For private registry repo, please make sure you are authorized to pull and push the target registry. The basic method is to use `docker pull` and `docker push` to verify your access to the source or target registry.

Expand Down
136 changes: 82 additions & 54 deletions docs/nydusd.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ We are working on enabling cloud-hypervisor support for nydus.

#### Use Different Storage Backends

Using different storage backend means that the nydus image metadata (bootstrap) layer is stored in the image registry, but the data layer will be stored on the external storage. Therefore, the option `--target` for `nydusify convert` is still required, the registry image reference is needed to store the metadata layer.

##### Localfs Backend

```
Expand Down Expand Up @@ -192,7 +194,7 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/
}
```

##### OSS backend with blobcache
##### OSS Backend

```
{
Expand All @@ -204,7 +206,32 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/
"endpoint": "region.aliyuncs.com",
"access_key_id": "",
"access_key_secret": "",
"bucket_name": ""
"bucket_name": "",
"object_prefix": "nydus/"
}
},
...
},
...
}
```

##### S3 Backend

```
{
"device": {
"backend": {
"type": "s3",
"config": {
...
"endpoint": "s3.amazonaws.com",
"scheme": "https",
"access_key_id": "",
"access_key_secret": "",
"bucket_name": "",
"region": "",
"object_prefix": "nydus/"
}
},
...
Expand All @@ -213,7 +240,7 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/
}
```

##### Registry backend
##### Registry Backend

```
{
Expand Down Expand Up @@ -246,41 +273,51 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/
```
Note: The value of `device.backend.config.auth` will be overwrite if running the nydusd with environment variable `IMAGE_PULL_AUTH`.

##### Enable P2P Proxy for Storage Backend
#### HTTP Proxy Backend

Add `device.backend.config.proxy` field to enable HTTP proxy for storage backend. For example, use P2P distribution service to reduce network workload and latency in large scale container cluster using [Dragonfly](https://d7y.io/) (enable centralized dfdaemon mode).
The `HttpProxy` backend can access blobs through a http proxy server which can be local (using unix socket) or remote (using `https://` or using `http://`).

`HttpProxy` uses two API endpoints to access the blobs:
- `HEAD /path/to/blobs` to get the blob size
- `GET /path/to/blobs` to read the blob

The http proxy server should respect [the `Range` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-range) to compute the offset and length of the blob.

The example config files for the `HttpProxy` backend may be:

```
// for remote usage
{
"device": {
"backend": {
"type": "registry",
"backend": {
"type": "http-proxy",
"config": {
"proxy": {
// Access remote storage backend via P2P proxy, e.g. Dragonfly dfdaemon server URL
"url": "http://p2p-proxy:65001",
// Fallback to remote storage backend if P2P proxy ping failed
"fallback": true,
// Endpoint of P2P proxy health checking
"ping_url": "http://p2p-proxy:40901/server/ping",
// Interval of P2P proxy health checking, in seconds
"check_interval": 5
},
...
"addr": "http://127.0.0.1:9977",
"path": "/namespace/<repo>/blobs"
}
},
...
},
...
}
}
}
```

Once the configuration is loaded successfully on nydusd starting, we will see the log as shown below:
or

```
INFO [storage/src/backend/connection.rs:136] backend config: CommonConfig { proxy: ProxyConfig { url: "http://p2p-proxy:65001", ping_url: "http://p2p-proxy:40901/server/ping", fallback: true, check_interval: 5 }, timeout: 5, connect_timeout: 5, retry_limit: 0 }
// for local usage
{
"device": {
"backend": {
"type": "http-proxy",
"config": {
"addr": "/path/to/unix.sock",
}
}
}
}
```

The `HttpProxy` backend also supports the `Proxy` and `Mirrors` configurations for remote usage like the `Registry backend` described above.

##### Enable Mirrors for Storage Backend (Recommend)

Nydus is deeply integrated with [Dragonfly](https://d7y.io/) P2P mirror mode, please refer the [doc](https://d7y.io/docs/setup/integration/nydus) to learn how configuring Nydus to use Dragonfly.
Expand Down Expand Up @@ -332,51 +369,42 @@ Currently, the mirror mode is only tested in the registry backend, and in theory
}
```

#### HTTP proxy backend

The `HttpProxy` backend can access blobs through a http proxy server which can be local (using unix socket) or remote (using `https://` or using `http://`).

`HttpProxy` uses two API endpoints to access the blobs:
- `HEAD /path/to/blobs` to get the blob size
- `GET /path/to/blobs` to read the blob

The http proxy server should respect [the `Range` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-range) to compute the offset and length of the blob.
##### Enable P2P Proxy for Storage Backend

The example config files for the `HttpProxy` backend may be:
Add `device.backend.config.proxy` field to enable HTTP proxy for storage backend. For example, use P2P distribution service to reduce network workload and latency in large scale container cluster using [Dragonfly](https://d7y.io/) (enable centralized dfdaemon mode).

```
// for remote usage
{
"device": {
"backend": {
"type": "http-proxy",
"backend": {
"type": "registry",
"config": {
"addr": "http://127.0.0.1:9977",
"path": "/namespace/<repo>/blobs"
"proxy": {
// Access remote storage backend via P2P proxy, e.g. Dragonfly dfdaemon server URL
"url": "http://p2p-proxy:65001",
// Fallback to remote storage backend if P2P proxy ping failed
"fallback": true,
// Endpoint of P2P proxy health checking
"ping_url": "http://p2p-proxy:40901/server/ping",
// Interval of P2P proxy health checking, in seconds
"check_interval": 5
},
...
}
}
}
},
...
},
...
}
```

or
Once the configuration is loaded successfully on nydusd starting, we will see the log as shown below:

```
// for local usage
{
"device": {
"backend": {
"type": "http-proxy",
"config": {
"addr": "/path/to/unix.sock",
}
}
}
}
INFO [storage/src/backend/connection.rs:136] backend config: CommonConfig { proxy: ProxyConfig { url: "http://p2p-proxy:65001", ping_url: "http://p2p-proxy:40901/server/ping", fallback: true, check_interval: 5 }, timeout: 5, connect_timeout: 5, retry_limit: 0 }
```

The `HttpProxy` backend also supports the `Proxy` and `Mirrors` configurations for remote usage like the `Registry backend` described above.

### Mount Bootstrap Via API

To mount a bootstrap via api, first launch nydusd without a bootstrap:
Expand Down

0 comments on commit b08efb1

Please sign in to comment.