Support for exporting nydus compression type

[imeoer]

Related issue: #2046

Nydus image is a container accelerated image format provided by the Dragonfly image-service project, which offers the ability to pull image data on demand, without waiting for the entire image pull to complete and then start the container. It has been put in production usage and shown vast improvements over the OCI v1 image format in terms of container launching speed, image space, and network bandwidth efficiency, as well as data integrity. Nydus image can be flexibly configured as a FUSE-based user-space filesystem or in-kernel FS with nydus daemon in user-space:

• Lightweight integration with VM-based containers runtime like KataContainers. KataContainers is supported Nydus as a native image acceleration solution.
• Nydus closely cooperates with Linux in-kernel disk filesystem containers' rootfs can directly be set up by EROFS over Fscache with lazy pulling capability. The corresponding changes had been merged into Linux kernel since v5.19.

Nydus has provided a conversion tool Nydusify for converting OCIv1 image to Nydus image and integrated into Harbor Acceld as a conversion driver, which assumes that the OCI image is already available in the registry, but a better way would be to build the Nydus images directly from the build system instead of using the conversion tool, which would increase the speed of the image export, so we experimentally integrated the Nydus export in Buildkit.

A manifest example of nydus image can be found here. Unlike other compression formats (gzip, zstd, eStargz, etc.) in OCI image, nydus is divided into two types of layer, blob and bootstrap, where blob serves as the data and metadata for all files of each layer of the image, and bootstrap serves as the metadata of the whole image, the bootstrap is equivalent to the view of the whole image filesystem after all layers overlay, nydus use this layer form to reduce the number of HTTP request and FUSE overlay's overhead on preparing the container's rootfs. For example, for an OCI image with 3 layers, the corresponding nydus image is 4 layers (3 layers of blob + 1 layer of bootstrap).

The converter package in nydus-snapshotter do the actual layer compression, this PR imports the package to implement the export of nydus compression type.

[imeoer]

@ktock @AkihiroSuda PTAL

[ktock]

BuildKit doesn't seem to be able to use the exported nydus-compressed image and cache.
e.g. the second build in the following with inline cache panics.

# mkdir -p /tmp/ctx
cat <<EOF > /tmp/ctx/Dockerfile
FROM registry2-buildkit:5000/ubuntu:20.04
RUN echo hello > /hello
EOF
# buildctl build --progress=plain --frontend=dockerfile.v0 --local context=/tmp/ctx --local dockerfile=/tmp/ctx \
                 --output type=image,name=registry2-buildkit:5000/ubuntu:20.04-hello,push=true,compression=nydus,oci-mediatypes=true \
                 --export-cache type=inline
# buildctl build --progress=plain --frontend=dockerfile.v0 --local context=/tmp/ctx --local dockerfile=/tmp/ctx \
                 --output type=image,name=registry2-buildkit:5000/ubuntu:20.04-hello-zstd,push=true,compression=zstd,oci-mediatypes=true \
                 --import-cache type=registry,ref=registry2-buildkit:5000/ubuntu:20.04-hello
...
panic: runtime error: index out of range [4] with length 4

goroutine 748 [running]:
github.com/moby/buildkit/cache/remotecache.(*contentCacheImporter).importInlineCache.func1.1()
	/go/src/github.com/moby/buildkit/cache/remotecache/import.go:188 +0xda5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/src/github.com/moby/buildkit/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/moby/buildkit/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x92

[imeoer]

BuildKit doesn't seem to be able to use the exported nydus-compressed image and cache. e.g. the second build in the following with inline cache panics.

@ktock Thanks for the testing, the panic indeed occurs.

This seems to be a tricky problem. Nydus image consists of multiple blobs and a bootstrap (one more layer than the cache records), and the blob does not correspond to the cache record one by one (probably fewer layers than the cache record).

Is it possible to consider not supporting the inline cache export for nydus image? Will submit a new commit to do this.

[imeoer]

@ktock rebased code to resolve the conflicts with the PR and add workaround for inline export on the commit. PTAL.

[ktock]

Now we seem to have 2 implementation of computeBlobChain(computeBlobChain and computeNydusBlobChain).
Some concerns about maintainability:

• How can guarantee/keep the compatibility of these implementations?
• Will we need 2 duplicated patches on these implementations for 1 futural change?

[bergwolf]

@AkihiroSuda @ktock @tonistiigi With #3136 merged, this PR is pretty clean in terms of code invasion in the general code path. Could you give it another spin and see if it is suitable for merging? As discussed earlier, we'll continue improving the nydus functionality in buildkit and we look forward to putting it into large-scale production use.

[imeoer]

@ktock @AkihiroSuda Updated, please help to take a glance again, thanks!

[imeoer]

Thanks for all! cc @tonistiigi Please help to take a final look, thanks!

[hsiangkao]

Thanks all for considering Nydus!