Mounting containerd filesystem on k3s Alpine

[jiayihu]

K3s on Alpine is giving the following error when mounting a fs for a container. Krustlet is not having the issue when mounting volumes.

failed: rpc error: code = Unknown desc = failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/29/work upperdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/29/fs lowerdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs]}: invalid argument: unknown

[jiayihu]

K3s doesn't seem to work nicely with Overlay filesystem: k3s-io/k3s#19 (comment)

On the other hand, it seems the error happens also with rootless k3s and the support for rootless has improved in latest k3s. But latest k3s isn't working #36

Trying --snapshotter native also requires latest k3s

[jiayihu]

The issue is also likely caused by the overlay filesystem not being ext k3s-io/k3s#495 (comment)

[jiayihu]

Updating to v1.19.5+k3s2 and using --snapshotter native works, although there is now the issue invalid capacity 0 on image filesystem, which presented even back with Ubuntu Server. Probably caused by the slow SD cards?

E1227 21:05:03.699917    2384 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.native": unable to find data in memory cache.
E1227 21:05:03.700114    2384 kubelet.go:1218] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem

[jiayihu]

Fixed by reinstalling Alpine with persistent ext fs since nested overlays don't play nice with containerd k3s-io/k3s#2755