Nydus Image Service v2.0.0

Highlights

• New on-disk image format RAFSv6 that is compatible with the Linux kernel EROFS file system.
• New nydus-backend-proxy to serve a local directory as a nydusd blob backend storage.
• New nydus-overlayfs command to help containerd to mount a nydus format snapshotter mounts.
• nydusd supports the Darwin platform and presents the Nydus image to Darwin users via FUSE.
• nydusd supports dynamic read amplification to better utilize backend storage bandwidth.
• nydusify supports pack subcommand to convert a local directory into nydus bootstrap and blob.
• nydusify supports --chuck-dict option to use a predefined bootstrap as a chunk deduplication hint.
• nydus-image supports merging two nydus bootstrap into a single bootstrap.
• nydus-image supports exporting a tar-formated blob containing a layered bootstrap and a data chunk blob.
• nydus-snapshotter has been contributed to the containerd community as a sub project.
• Various bugfixes and doc improvements.

What's Changed

• makefile: a option to build golang components without docker by @changweige in #213
• vendor: update fuse-backend-rs dependency by @bergwolf in #214
• snapshotter: don't touch original nydusd auth if no auth in labels by @changweige in #212
• cargo: use event-manger from crate.io by @bergwolf in #215
• action: build all static binaries and include them in release tarball by @bergwolf in #216
• build(deps): bump github.com/opencontainers/image-spec from 1.0.1 to 1.0.2 in /contrib/nydusify by @dependabot in #218
• build(deps): bump github.com/containerd/containerd from 1.4.11 to 1.4.12 in /contrib/nydus-snapshotter by @dependabot in #220
• release: include an example nydusd config in the release tarball by @bergwolf in #217
• build(deps): bump github.com/containerd/containerd from 1.4.11 to 1.4.12 in /contrib/nydusify by @dependabot in #221
• build(deps): bump github.com/containerd/containerd from 1.5.7 to 1.5.8 in /contrib/ctr-remote by @dependabot in #222
• build(deps): bump github.com/opencontainers/image-spec from 1.0.1 to 1.0.2 in /contrib/ctr-remote by @dependabot in #224
• rafs: fix up access API root mode by @changweige in #231
• Introduce static check as a restrict step when building snapshotter and silent warnings by @changweige in #229
• docs: approach to start container by nerdctl and refine docs by @changweige in #230
• improve passthroughfs when running together with rafs by @bergwolf in #232
• nydusd: add fusedev writable cmd argument by @bergwolf in #233
• Introduce nydus-v2.0/rafsv6 by @jiangliu in #234
• docs: fix some typos and grammatical problem by @liubin in #235
• app: fix incorrect crate name in README.md by @liubin in #236
• Doc: update performance diagram by @imeoer in #240
• Doc: update performance diagram by @imeoer in #241
• RAFS V6 update by @hsiangkao in #242
• nydus-snapshotter: fix recreate daemon config/socket dir by @luodw in #243
• cache: set the number of worker threads of tokio threads pool by @changweige in #246
• Nydusd: fix panic on cli boot by @imeoer in #248
• cargo: update fuse-backend-rs dependency by @bergwolf in #249
• A series of changes for supporting diff build cache by @imeoer in #250
• snapshotter: pass auth from docker config by @bergwolf in #251
• fix ctr remote build and add CI by @bergwolf in #253
• workflow: add workflow to convert top docker hub images by @bergwolf in #254
• readme: refine the description of #community by @hsiangkao in #260
• Utilize golangci-lint to perform static check for nydus-overlayfs by @changweige in #256
• add golangci-lint tool for ctr-remote tool by @changweige in #255
• nydusify: utilize golangci-lint to perform static check by @changweige in #258
• readme: add brief introduction for nydus-overlayfs by @changweige in #257
• Add golangci lint graphdriver by @changweige in #261
• misc: update top image list by @bergwolf in #262
• snapshotter: bump version of github.com/opencontainers/image-spec to v1.0.2 by @changweige in #266
• workflow: fix up cron time by @bergwolf in #271
• Builder: fix no bootstrap output in v6 diff build by @imeoer in #272
• build(deps): bump github.com/containerd/containerd from 1.5.8 to 1.5.9 in /contrib/ctr-remote by @dependabot in #273
• Bump version to 2.0.0-rc.0 by @imeoer in #280
• Enable release workflow for RC version by @imeoer in #282
• fix some contrib projects depbot warnings by @bergwolf in #283
• nydusd: fix thread-num validation by @liubin in #287
• Migrate snapshotter to containerd org by @changweige in #270
• A couple of fixes for Rafs v6 disk layout by @changweige in #288
• Rafs: validate metadata stored in bootstrap by @changweige in #289
• Implement Rafs v6 for nydusd by @changweige in #290
• ctr-remote: pin image-spec and containerd versions by @bergwolf in #293
• nydusd-virtiofs: Implement set_state/get_state to avoid panicking by @liubin in #295
• action: add cargo deny support by @bergwolf in #292
• docs: containerd configuration version 2 setting by @luodw in #297
• cargo: update fuse-backend-rs dependency by @bergwolf in #298
• chunk-dict: fix load chunk-dict err by @hhhhsdxxxx in #301
• nydus-image: (experimetal)support compact blob by @hhhhsdxxxx in #302
• nydusify: add reference blob layers to manifests by @hhhhsdxxxx in #303
• rafs: rename is_all_chunk_ready to all_chunks_ready by @changweige in #305
• improve docs by @hhhhsdxxxx in #307
• nydusify support pack subcommand by @hhhhsdxxxx in #306
• test: use is_ok/is_err replace match for Result return type by @liubin in #310
• build(deps): bump github.com/docker/distribution from 2.7.1+incompatible to 2.8.0+incompatible in /contrib/nydusify by @dependabot in #311
• cache: also commit current region for internal IO by @changweige in #308
• Prepare for publishing nydus-utils crate by @jiangliu in #313
• nydusify: support compact blob before pack by @hhhhsdxxxx in #309
• cache: return Ok(false) when io inflight tracer timeouts by @changweige in #314
• cache: commit region for internal IO if hits gzip format by @changweige in #315
• rafs: only append more chunks to last bio when amplify io by @changweige in #317
• Sync missing commits by @bergwolf in #320
• Add capability of prefetch for rafs v6 format by @changweige in #319
• readme: add ci status badget by @bergwolf in #321
• build(deps): bump github.com/containerd/containerd from 1.4.12 to 1.4.13 in /contrib/nydusify by @dependabot in #322
• Builder: support chunk dict in diff build by @imeoer in #323
• nydus-image: fix wrongly calculated prefetch table size by @changweige in #324
• Doc: add nydus ecosystem list by @imeoer in #326
• Fix some doc and add make clean target. by @liubin in #330
• Misc: unify go version to 1.17 by @imeoer in #328
• fix up release action by @bergwolf in #327
• Makefile: add missed make command for build_golang by @liubin in #331
• Some minor fixes by @liubin in #332
• Doc: add p2p proxy field in nydusd config by @imeoer in #333
• bugfix: storage: Fix the race problem when decompressing with lz4 by @MercyMM in #335
• nydusd: fix is_mount failure by @liubogithub in #339
• cargo: update openssl-src dependency by @bergwolf in #336
• fix up UT by @bergwolf in #337
• refact: use mio for daemon exit signal monitoring by @ccx1024cc in #340
• contrib: update dependency to fix security issues by @bergwolf in #341
• nydus-graphdriver: use fixed docker/distribution version by @bergwolf in #342
• doc: fix cargo doc issues by @liubin in #345
• contrib: add nydusd-http-server by @nan1994 in #304
• nydusify: update docker/distribution dependency by @nan1994 in #347
• Nydus image by @MercyMM in #350
• storage: update BlobPrefetchRequest fields type from u32 to u64 by @MercyMM in #349
• fix v6 panic due to a trace! by @liubogithub in #346
• storage: refactor file cache code for coming fscache by @jiangliu in #353
• Nydusify: pipe stdin to builder if enable --prefetch-patterns by @imeoer in #344
• makefile: correct cargo clippy by @bergwolf in #357
• blobfs: ensure blob cache dir exist by @MercyMM in #358
• refactor: port to macos by @killagu in #316
• Refine nydusd commandline parsing code by @jiangliu in #361
• Refactor the nydus-api crate to prepare for API v2 by @jiangliu in #364
• makefile: fix error handling and always run virtiofs ut by @bergwolf in #365
• util: move compress from storage into nydus-utils by @jiangliu in #369
• nydusd: handle api server mio poll error by @bergwolf in #371
• Builder: make repeatable for root directory by @imeoer in #372
• Nydusify: ignore arch for single manifest by @imeoer in #359
• Builder: support tar build by @imeoer in #351
• doc: add macos doc by @killagu in #368
• test remote::server::tests::test_reconnect: Give 10 secs more to try by @hsiangkao in #375
• action: use softprops/action-gh-release to do release by @bergwolf in #379
• docker-nydus-graphdriver: avoid `overlayfs: overlapping path' by @hsiangkao in #380
• Backend: support skip_verify option by @imeoer in #382
• Builder: remove has_xattr flag check for merge by @imeoer in #383
• http: fix waker dead improperly due to lifetime mismatch by @hsiangkao in #386
• Add support for zstd compression algorithm for blob file. by @pigletDips in #388
• nydusd: graceful state machine thread shutdown by @bergwolf in #384
• Use Caret to specify dependency versions. by @wllenyj in #398
• add global log level option for nydusify by @liubin in #400
• prepare for 2.0.0 release by @bergwolf in #399
• fix: not amplify io if next file in different blob by @killagu in #406
• Cargo: update fuse-backend-rs to 0.9.0 by @wllenyj in #408
• Fix some nydus-image inspect issues by @liubin in #409
• action: support multiple arch releases by @imeoer in #407
• Fix convert and release workflow issues by @bergwolf in #410
• Cargo: update crossbeam-utils dependency by @bergwolf in #411
• Bug fix: the .. disappear. by @pigletDips in #405
• nydus-image: Feature, add a block allocator for reducing bootstrap size by reusing used blocks by @pigletDips in #412
• builder: implement tar like build by @imeoer in #415
• builder: fix lost hardlink by @imeoer in #413
• nydusify: change args order when calling nydus-image by @liubin in #420
• cat heredoc will not work with sudo by @wanderxjtu in #418
• cargo: virtiofs does not depend on blobfs by @bergwolf in #422
• action: release macos bianries as well by @bergwolf in #429
• rafs: delete RafsSuperBlobs trait by @liubin in #421
• rafs v6: bug fix for the incorrect dirents process by @pigletDips in #431
• action: add shasum file for released assets by @imeoer in #430
• test(nydus-image): remove unneeded tree dependency in unit test by @pigletDips in #433
• docs: fix some typo by @pigletDips in #434
• smoke: add basic testing for rafs v6 by @imeoer in #438
• storage: ignore two unstable tests by @bergwolf in #443
• docs: update brief summary about rafs filesystem by @hsiangkao in #444
• chore: update tutorial.md by @fatelei in #442
• contrib: update runc dependency by @bergwolf in #445

New Contributors

• @hhhhsdxxxx made their first contribution in #301
• @MercyMM made their first contribution in #335
• @ccx1024cc made their first contribution in #340
• @killagu made their first contribution in #316
• @pigletDips made their first contribution in #388
• @wllenyj made their first contribution in #398
• @wanderxjtu made their first contribution in #418
• @fatelei made their first contribution in #442

Full Changelog: v1.1.0...v2.0.0