CPU CFS quota fails if comounted controller order from /proc/self/cgroup does not match /sys/fs/cgroup

[brandond]

Environmental Info:
K3s Version:
N/A

Node(s) CPU architecture, OS, and Version:
N/A

Cluster Configuration:
N/A

Describe the bug:
Copied from rancher/rke2#689:

On CentOS7 (cgroups v1) RKE2 incorrectly disables CFS, starting kubelet with --cpu-cfs-quota=false. (The message "... level=warning msg="Disabling CPU quotas due to missing cpu.cfs_period_us" is observed in the RKE2 log; issued here). As a result kubelet does not configure CPU limit enforcement. This likely also affects K3S, and other 3.10.x Kernel -based distributions, but I have only tested with RKE2 (v1.19.7+rke2r1) / CentOS7 (3.10.0-1127.19.1.el7.x86_64).

The underlying problem is likely in K3S (where it has been diagnosed once, but not yet fixed). K3S/RKE2 checkCgroups logic in the agent, responsible for determining CFS capability, expects the relevant entry in /proc/{pid}/cgroup file to have systems part with cpu (e.g. cpu,cpuacct) that matches the name of folder in /sys/fs/cgroup. This is not the case in CentOS7 (and possibly other 3.10.0 -based distros?), where /proc/{pid}/cgroup files have cpuacct,cpu entry, but /sys/fs/cgroup has cpu,cpuacct folder (order reversed).

Steps To Reproduce:

• Installed K3s:

Expected behavior:
CFS quota is enabled

Actual behavior:
CFS quota is not enabled

Additional context / logs:

[brandond]

Not quite sure how to cause the controller order to be different between the two locations - I've never seen it myself.

[rancher-max]

In Centos7, I see the order being different on both the latest master commit (5749f66aa3c469b4c744caab1878c6967ac08041) and on v1.19.7+k3s1. In both cases I get the following:

$ grep cpu /proc/self/cgroup 
8:cpuset:/
2:cpuacct,cpu:/

$ ls -lad /sys/fs/cgroup/*cpu*
lrwxrwxrwx. 1 root root 11 Feb 16 20:50 /sys/fs/cgroup/cpu -> cpu,cpuacct
lrwxrwxrwx. 1 root root 11 Feb 16 20:50 /sys/fs/cgroup/cpuacct -> cpu,cpuacct
drwxr-xr-x. 4 root root  0 Feb 16 20:50 /sys/fs/cgroup/cpu,cpuacct
drwxr-xr-x. 3 root root  0 Feb 16 20:50 /sys/fs/cgroup/cpuset

However, on v1.19.7+k3s1 I see the warning message in the logs: level=warning msg="Disabling CPU quotas due to missing cpu.cfs_period_us", whereas on the latest master commit I do not see this. Furthermore, on v1.19.7 the kubelet starts with the following (notice --cpu-cfs-quota=false), whereas this is not the case (the argument is not present at all) on latest master commit:

level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/30740d1d67da51fe92b10367ecce4d580e552c634ad4a6c4dd13297ffd1f3edd/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=/run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --cpu-cfs-quota=false --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=[REDACTED] --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"

For the above reasons, I believe this is fixed in k3s and will be present in v1.20.3+k3s1 and later.