kind v0.10 does not work with Kubernetes v1.21

[spam-n-eggs]

What happened: When creating the default kind cluster, the docker container started but exited immediately
From kind export logs:

ERROR: [command "docker exec --privileged kind-control-plane sh -c 'tar --hard-dereference -C /var/log/ -chf - . || (r=$?; [ $r -eq 1 ] || exit $r)'" failed with error: exit status 1, [command "docker exec --privileged kind-control-plane journalctl --no-pager" failed with error: exit status 1, command "docker exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 1, command "docker exec --privileged kind-control-plane journalctl --no-pager -u kubelet.service" failed with error: exit status 1, command "docker exec --privileged kind-control-plane journalctl --no-pager -u containerd.service" failed with error: exit status 1]]

What you expected to happen: The control plane to start and create the kubernetes cluster

How to reproduce it (as minimally and precisely as possible): On a fresh install of Alpine 3.13, with docker installed, download the latest version of kind via go get, and run kind create cluster

Anything else we need to know?: Could this be the same issue as #2228 ?

Environment:

• kind version: (use kind version): kind v0.10.0 go1.15.10 linux/amd64
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info):

Client:
 Context:    default
 Debug Mode: false
Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 3
 Server Version: 20.10.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: 
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.29-0-lts
 Operating System: Alpine Linux v3.13
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.3GiB
 Name: docker-manager
 ID: AAYK:I4NF:BFL5:6C6R:76QU:3A7X:J6BL:ZJBO:WK7H:2D4V:VUUO:3HXF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release):

NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.13.5
PRETTY_NAME="Alpine Linux v3.13"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

[BenTheElder]

this is probably because of alpine using openrc instead of systemd. that should already be fixed at HEAD by #2091

[spam-n-eggs]

Thanks @BenTheElder. I updated to the main branch and it's working without a problem. Any Idea on when that is going to make it into a release?
For anyone else that stumbles upon this issue, the go get command I used is: GO111MODULE="on" go get sigs.k8s.io/kind@main

[BenTheElder]

O(soon) ™️, mostly want to finish #2176

[lazedo]

it works with 1d4788d kubernetes version v1.20
fails with 247fea5a kubernetes version v1.21

al-01:~# cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.13.5
PRETTY_NAME="Alpine Linux v3.13"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

al-01:~# docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version:
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.36-0-lts
 Operating System: Alpine Linux v3.13
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 3.842GiB
 Name: al-01
 ID: Q6XN:XZOL:FQUC:7G4Z:PEJS:RS5H:32JA:HQK6:ZVX4:RBIW:JWVR:LUGL
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio weight support
WARNING: No blkio weight_device support

May 14 16:38:26 kind-control-plane kubelet[1741]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 14 16:38:26 kind-control-plane kubelet[1741]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 14 16:38:26 kind-control-plane kubelet[1741]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 14 16:38:26 kind-control-plane kubelet[1741]: Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 14 16:38:26 kind-control-plane kubelet[1741]: I0514 16:38:26.503483    1741 server.go:197] "Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead"
May 14 16:38:26 kind-control-plane kubelet[1741]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 14 16:38:26 kind-control-plane kubelet[1741]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 14 16:38:26 kind-control-plane kubelet[1741]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 14 16:38:26 kind-control-plane kubelet[1741]: Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 14 16:38:26 kind-control-plane kubelet[1741]: I0514 16:38:26.519983    1741 server.go:440] "Kubelet version" kubeletVersion="v1.21.1"
May 14 16:38:26 kind-control-plane kubelet[1741]: I0514 16:38:26.520638    1741 server.go:851] "Client rotation is on, will bootstrap in background"
May 14 16:38:26 kind-control-plane kubelet[1741]: I0514 16:38:26.525657    1741 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
May 14 16:38:26 kind-control-plane kubelet[1741]: I0514 16:38:26.527878    1741 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
May 14 16:38:26 kind-control-plane kubelet[1741]: I0514 16:38:26.527932    1741 container_manager_linux.go:995] "CPUAccounting not enabled for process" pid=1741
May 14 16:38:26 kind-control-plane kubelet[1741]: I0514 16:38:26.527972    1741 container_manager_linux.go:998] "MemoryAccounting not enabled for process" pid=1741
May 14 16:38:31 kind-control-plane kubelet[1741]: E0514 16:38:31.544724    1741 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: invalid configuration: cgroup-root [\"kubelet\"] doesn't exist"
May 14 16:38:31 kind-control-plane systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE

al-01:~# docker logs kind-control-plane
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v1
INFO: fix cgroup mounts for all subsystems
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: faking /sys/class/dmi/id/product_uuid to be random
INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
INFO: setting iptables to detected mode: legacy
INFO: Detected IPv4 address: 172.18.0.2
INFO: Detected IPv6 address: fc00:f853:ccd:e793::2
systemd 246.6-1ubuntu1.3 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture x86-64.
Failed to create symlink /sys/fs/cgroup/cpuacct: File exists
Failed to create symlink /sys/fs/cgroup/cpu: File exists
Failed to create symlink /sys/fs/cgroup/net_prio: File exists
Failed to create symlink /sys/fs/cgroup/net_cls: File exists

Welcome to Ubuntu 20.10!

Set hostname to <kind-control-plane>.
Couldn't move remaining userspace processes, ignoring: Input/output error
Queued start job for default target Graphical Interface.

[BenTheElder]

We know that the cgroups fix worked on a different systemdless distro from another user but not this one. We'll need more data.

If you run a privileged container and dump the mounts and cgroups tree what do you get?

[BenTheElder]

@lazedo please file another bug with your info. I'm going to close this one as we solved OPs problem for the original configuration provided.

[BenTheElder]

Ahhh rereading this it's the same distro same version just a newer kind commit / kubernetes. My apologies. We frequently have issues fragmenting into 10 different actual problems from different users woven into the same issue and I've gotten too quick to break them up.

[BenTheElder]

One more here #166 (comment) but we don't know anything about the host other than probably arm64.

[lazedo]

@BenTheElder the host in my test for alpine 3.13.5 is amd64 (actually a vm inside hyper-v but worked until now). other previleged containers work such as dind.

[BenTheElder]

Interesting thanks, dind is slightly slightly simpler as it doesn't need to make kubelet happy.

If you run a privileged container and dump the mounts and cgroups tree what do you get?

We need some info like this to determine the mount structure being used, this sounds wrong. Eg. docker should not be mounting /sys/fs/cgroup/*. But since we, and none of the CI for kubernetes, containerd etc. use openrc, we're not sure. Everything integrates with systemd, but we're doing our best here.

you can see more about what we need to do here

kind/images/base/files/usr/local/bin/entrypoint

Line 175 in 247fea5

fix_cgroup() {

[lasdolphin]

Same issue here:
Docker version 20.10.6, build 370c289
masOS BigSur 11.3
MacbookAir M1
Everything works with alternative node image rossgeorgiev/kind-node-arm64

EDIT: added docker info output

 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.942GiB
 Name: docker-desktop
 ID: 66ZC:ZG4M:TEB2:7TN2:6GXG:AA7X:IQIX:EPVO:BHQR:XJKP:MJZR:JJPC
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[BenTheElder]

@lasdolphin I think you were using v1.21 image with kind v0.10.0? that's a different problem and isn't expected to work and will be fixed by upgrading to v0.11 #166 (comment) which is release-in-progress as I type.

#2236 (comment) also suggests that is what @spam-n-eggs's issue was.

@lazedo were you also trying to run Kubernetes v1.21.0+ with kind v0.10? That does not work due to a breaking change in kubernetes/kubeadm, it's fixed in the latest kind sources which are receiving a tagged release currently.

If not: please file a new issue with your details.

[BenTheElder]

#2189

[lazedo]

@BenTheElder i was running kind from HEAD

[BenTheElder]

https://github.com/kubernetes-sigs/kind/releases/tag/v0.11.0#contributors is out

@lazedo I think we need a separate issue to track your problem as we fixed the original here and it's not clear if they were related, but we should cross reference them.