Fails to start 1.21.0 cluster - kubelet complains about missing cgroup #2189
Comments
|
#2188 (comment) response here. This is an intentional breakage upstream in Kubernetes and is not any of the (good!) guesses here, it relates to the container runtime and kubelet mismatching drivers because kubeadm made a breaking change in v1.21 |
|
KIND in Kubernetes presubmit but using kind @ HEAD because Kubernetes can and will make breaking changes like this and we have to keep up. |
|
should it be mentioned in https://kind.sigs.k8s.io/docs/user/quick-start/#creating-a-cluster that i see this as a re-occurring problem from users. |
|
Well I'd like to stop letting this happen, it is relatively rare that most components have breaking changes versus users moving to cgroupsv2 etc., but it does seem to be technically allowed in upstream components .. I think Kubernetes should stop doing this and not have single release action-required changes, as it's a bad experience for people trying to run it themselves. Most things like the end user facing APIs do not do this. |
it is a multi-layered problem for a WIP platform that is not versioned properly. not following SemVer and doing breaking changes on MINORs post a v1.x is definitely something that i do not agree with.
users are still forced to use tools like |
|
Not every k8s release has a corresponding kind release, so it's not possible for a user to know up-front whether a new k8s will work an existing kind or not. I'm not sure if the SIGs can coordinate to ensure the release of kind is synchronous with the release of k8s in case of breaking changes. If not, either users like me are going to open issues for tracking, or if you know up-front that breakage will happen you can create a pinned issue for it yourself. I only opened my issue after not finding an existing one. |
|
documenting a support matrix can help. |
|
For completeness, I should note that my distro added support for cgroups v2, and I can see the cgroup being created via the |
|
Fixed with the 0.11.0 release. I was successfully able to deploy a 1.21.1 cluster. |
What happened:
kind create cluster --name v1.21 --image 'kindest/node:v1.21'The
kindcommand fails with:Click here
journalctlinside the container reveals:What you expected to happen:
Expected it to succeed.
How to reproduce it (as minimally and precisely as possible):
As above.
Anything else we need to know?:
This works with other Kubernetes versions from 1.14 through 1.20, so this is something 1.21-specific. The
--cgroup-root=/kubeletparameter is given to kubelet for these versions too, without problems.Environment:
kind version): kind v0.10.0 go1.15.7 linux/amd64kubectl version): 1.21.0docker info): Server Version: 20.10.5-ce/etc/os-release): openSUSE Tumbleweed 20210330I guess v1.21 changed to require cgroups v2? That's the only thing that makes sense.If that's the case, then yes my distro doesn't have the file it's looking for. The kubelet service unit has anExecStartPre(injected via/etc/systemd/system/kubelet.service.d/10-kubeadm.conf):which means the script is entirely skipped on cgroup v1 hosts. If the script was run, it would've produced a more descriptive error:If v1.21 requires cgroups v2, it would be useful to run that script unguarded so that the error can be seen in the journal.My distro added support for cgroups v2, and I can see the
kubeletcgroup being created, but kubelet still exits with the same error that the cgroup is missing.$ docker exec -it v1.21-control-plane cat /sys/fs/cgroup/kubelet/cgroup.controllers cpuset cpu io memory hugetlb pids rdmaThe text was updated successfully, but these errors were encountered: