support default-cgroupns-mode=private

[acuteaura]

Description

Private CGroup modes are required for some (privileged) workloads, like Cilium's eBPF kube-proxy replacement:

Official documentation: https://docs.cilium.io/en/v1.13/installation/kind/#install-cilium (in the large notice box)

Related issue at Cilium: cilium/cilium#25479

Setting default-cgroupns-mode: private inside the docker key in colima's config leads to runc crash:

❯ docker run -it --rm alpine ls -la /proc/self/ns/cgroup
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
08409d417260: Pull complete 
Digest: sha256:02bb6f428431fbc2809c5d1b41eab5a68350194fb508869a33cb1af4444c9b11
Status: Downloaded newer image for alpine:latest
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/7), flags: 0xf, data: openrc: invalid argument: unknown.

You can confirm cgroupv2 per-container slices by running the above command twice and confirming different IDs are assigned; colima without options produces the same ID.

❯ docker run -it --rm alpine ls -la /proc/self/ns/cgroup
lrwxrwxrwx    1 root     root             0 May 23 07:48 /proc/self/ns/cgroup -> cgroup:[4026531835]

❯ docker run -it --rm alpine ls -la /proc/self/ns/cgroup
lrwxrwxrwx    1 root     root             0 May 23 07:48 /proc/self/ns/cgroup -> cgroup:[4026531835]

[muvaf]

FYI, It's a showstopper for kind and Rancher Desktop as well kubernetes-sigs/kind#3277

[stenh0use]

+1 to this feature.

[the-gigi]

👍

[BenTheElder]

I think the more compelling argument is that related issues appear to be coming up with buildx #764 / rancher-sandbox/rancher-desktop#5363 that also appear to be related to the cgroups under alpine/openrc, not just with kind and --cgroupns=private.

It appears that cgroups are off under this distro and that should either be fixed or the distro switched.

AFAICT lima + debian,ubuntu VMs work fine.

[acuteaura]

I think the more compelling argument is that related issues appear to be coming up with buildx #764 / rancher-sandbox/rancher-desktop#5363 that also appear to be related to the cgroups under alpine/openrc, not just with kind and --cgroupns=private.

It appears that cgroups are off under this distro and that should either be fixed or the distro switched.

AFAICT lima + debian,ubuntu VMs work fine.

https://wiki.alpinelinux.org/wiki/OpenRC#cgroups_v2

[stenh0use]

@acuteaura I'm assuming that you posted that link to point to the documentation on how to enabled cgroupsv2? Have you tried enabling cgroups via the rc.conf file and also running those commands? I tried many things the other day and it doesn't seem possible.

[acuteaura]

I haven't had the time to spend debugging this speficially for Alpine, but looking at my notes when I did for Fedora (we were working on a reproducible kind+cilium+kubeproxyless mode setup), Docker does something quite specific to Ubuntu and is/was broken on Fedora with cgroup_no_v1=all as well. Mostly Docker seems to expect controllers to be mounted at /sys/fs/cgroup when most systems place it at /sys/fs/cgroup/unified. The fix for fedora was to use cgroup_no_v1=net_cls,net_prio and let it mount in hybrid mode or to mount cgroup2 over /sys/fs/cgroup.

I don't think Alpine is likely to blame for this and adjusting the mount point should let it run with pure cgroup2.

[stenh0use]

I haven't tried too hard, but the documented way to mount cgroup2 on /sys/fs/cgroup doesn't seem to work.

# "unified" mounts cgroups version 2 on /sys/fs/cgroup                        
rc_cgroup_mode="unified"   

My motivation for this is also to get cilium working with docker in docker, but with nomad rather than kind/k8s. I'd also be in support of having systemd based distro as mentioned in #369 as I'd like to be able to run dind with docker setting native.cgroupdriver=systemd.

[BenTheElder]

Mostly Docker seems to expect controllers to be mounted at /sys/fs/cgroup

This is the standard mount point for v1* and v2. It's not Ubuntu specific. In v2 only mode they should be mounted here as well.

/sys/fs/cgroup/unified should be v2 cgroups in mixed v1+v2 "hybrid" mode which is not recommended by pretty much any project.

• v1 it will actually be a dir with controllers mounted underneath but.

e.g. RHEL also documents v2-only at /sys/fs/cgroup https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/using-cgroups-v2-to-control-distribution-of-cpu-time-for-applications_managing-monitoring-and-updating-the-kernel

[Cryptophobia]

I do not believe this is a feature at this point but a bug/issue. It has come to affect two things for me when working with colima that I have recently run into:

1. Working with latest version of KinD
   Solution: Downgrade to an earlier version of KinD (this solves the cgroups mounting issue)

2. Working with docker buildx
   Issue thread: ERROR: failed to solve: process "/bin/sh docker/buildx#1986 (comment)
   Confirmed issue with colima: ERROR: failed to solve: process "/bin/sh docker/buildx#1986 (comment)

Right now the temporary solution for 2 above is to use docker build instead of docker buildx build which is pretty inconvenient.

[Cryptophobia]

Various users are switching from colima to other docker environment solutions in order to avoid these errors. Similar issue threads like these: dagger/dagger#5593

[bm1216]

+1

[Cryptophobia]

There is a workaround for this I discovered in another thread related to this issue:

If you are on colima, a solution is to use rootless buildx builder:

docker buildx create \
              --name fixed_builder \
              --driver-opt 'image=moby/buildkit:v0.12.1-rootless' \
              --bootstrap --use

Before running the above command, can remove the older buildx runners with docker buildx rm ...

Solution from here:
#764 (comment)

Using a rootless buildx builder is a good security practice anyways.