Refine tarfs overall design and implementation details

[jiangliu]

This PR is based on #497 , and improve tarfs by:

• move loopdev setup from Prepare()/Merge() into Mount(), preparing for support of nodev mode in addition to blockdev.
• store tar file in the global cache directory, mount tarfs at "mnt" instead of "tarfs", to keep consistence with fusedev/fscache drivers.
• use tempfile/rename mechanism to easy error recover
• use lock to avoid some possible race conditions
• use mode 0750 instead of 0755 for new directories
• associate a Manager with filesystems managed by blockdev

[codecov]

Codecov Report

Merging #499 (f3c1541) into main (33bca86) will decrease coverage by 0.13%.
The diff coverage is 0.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #499      +/-   ##
==========================================
- Coverage   37.68%   37.55%   -0.13%     
==========================================
  Files          60       60              
  Lines        7115     7139      +24     
==========================================
  Hits         2681     2681              
- Misses       4122     4146      +24     
  Partials      312      312              

Files Changed	Coverage Δ
config/config.go	31.38% <ø> (ø)
config/global.go	24.27% <0.00%> (-7.38%)	⬇️
pkg/manager/manager.go	14.88% <0.00%> (ø)

[jiangliu]

With this PR applied:

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# nerdctl run --rm -ti --snapshotter nydus busybox
docker.io/library/busybox:latest:                                                 resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:6e494387c901caf429c1bf77bd92fb82b33a68c0e19f6d1aa6a3ac8d27a7049d:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:1b0a26bd07a3d17473d8d8468bea84015e27f87124b283b91d781bce13f61370: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b539af69bc01c6c1c1eae5474a94b0abaab36b93c165c0cf30b7a0ab294135a3:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 5.6 s                                                                    total:  2.8 Ki (504.0 B/s)                                       
[2023-06-26 10:58:49.234445 +08:00] INFO RAFS features: COMPRESSION_NONE | HASH_SHA256 | EXPLICIT_UID_GID | TARTFS_MODE
/ # 

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# mount
/dev/loop13 on /var/lib/containerd-nydus/snapshots/2/mnt type erofs (ro,relatime,user_xattr,acl,cache_strategy=readaround)
root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# losetup 
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                                            DIO LOG-SEC
/dev/loop13
             0      0         0  0 /var/lib/containerd-nydus/snapshots/2/fs/image/image.boot              0     512
/dev/loop12
             0      0         0  0 /var/lib/containerd-nydus/cache/71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7 0     512

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# ls -l /var/lib/containerd-nydus/cache/
total 10812
-rw-r--r-- 1 root root 5092352 Jun 26 11:05 71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7
-rw-r--r-- 1 root root 5976064 Jun 26 11:05 71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7.image.disk

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# ls -l /var/lib/containerd-nydus/snapshots/1/fs/image/
total 48
-rw-r--r-- 1 root root 28672 Jun 26 11:05 image.boot
-rw-r--r-- 1 root root 28672 Jun 26 11:05 layer.boot

root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# losetup /dev/loop100 /var/lib/containerd-nydus/cache/71d064a1ac7d46bdcac82ea768aba4ebbe2a05ccbd3a4a82174c18cf51b67ab7.image.disk
root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# veritysetup open --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=512 --hash-block-size=4096 --data-blocks 10970 --hash-offset 5619712 /dev/loop100 erofs_verity /dev/loop100 618d2431d6799ea489c66b2cd94b578b570af0e43f3866bcc53320c8aceb1ec3
root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# mount -t erofs -o ro /dev/mapper/erofs_verity /mnt/
root@liujiang-ecs-dev:~/ws/nydus-snapshotter.git/cmd/containerd-nydus-grpc# ls /mnt/
bin  dev  etc  home  lib  lib64  root  tmp  usr  var

[jiangliu]

Configuration for above test:
vim /etc/nydus/config.toml

enable_tarfs = true
tarfs_hint = false
tarfs_max_concurrent_proc = 10
tarfs_export_mode = "image_block_with_verity"

[kevinXYin]

Tested with some top images on dockerhub, compare rootfs generated by tarfs and overlayfs snapshotter for runc containers, there is no difference.

Tested-By Xin Yin

[changweige]

As this PR is quite large, I tried to review it a few times but they were not easy for me. So please forgive us for the slowwww procedure of reviewing it. :-)

[imeoer]

With this PR applied:
...

Maybe add this to a formal doc for the tarfs feature.

[changweige]

The manager manages all the nydusd daemons which serve nydus image, but looks like tarfs does not need any nydusd daemon. Why there is a blockdev manager which never abord a nydusd daemon.

[jiangliu]

Though tarfs doesn't use daemon, it does use Manager to support:

1. cache space
2. recovery
3. metrics

[changweige]

I think we already have a dedicated cache manager which can work alone without a daemon manager.
As tarfs is an erofs mount, nydus-snapshotter does not have to recover any nydus daemon.
Tarfs is running as an Erofs mount without a nydusd daemon, I think there is no metric exported.

[changweige]

Thanks, maybe we need more work to do

[changweige]

LGTM~ Thanks