ERROR: failed to create cluster: failed to generate kubeadm config content (on Alpine Linux 3.13.5)

[bdutta]

What happened:
When running kind create cluster to create a k8s cluster, got the following error:

lphost1:~$ time kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼 
 ✓ Preparing nodes 📦  
 ✗ Writing configuration 📜 
ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 1
Command Output: Error response from daemon: Container ae0a3835495e1904dfa817a384021971c02992ecab07e518903c2616500a9710 is not running
Command exited with non-zero status 1
real	1m 58.45s
user	0m 0.47s
sys	0m 0.33s
alphost1:~$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

What you expected to happen:
Was expecting kind create cluster to successfully create a k8s cluster

How to reproduce it (as minimally and precisely as possible):
Run kind create cluster on a freshly installed Alpine Linux 3.13.5 KVM (qemu) VM instance, with latest docker, go installed (from Alpine package repositories). Docker is configured and known to work fine (for 'Hello World' docker image). Note that in my case, the reproduction is 100%. Attempted four times, and same error seen each time.

Anything else we need to know?:
Any chance this might be same issue as issue 1288 ?
If so, how much RAM is expected to be allocated ? This VM has 4096GB of RAM and 4 vCPUs (i5-8440). Swap has been disabled (for performance). Here's the memory status of this VM:

alphost1:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           3936         144        1719           0        2073        3560
Swap:             0           0           0

So, how does one follow the recommendation to start 2 nodes at a time (from the other issue, i.e. 1288) ?
Pointer to requisite documentation would be highly appreciated.

Environment:

• kind version: (use kind version): kind v0.10.0 go1.15.10 linux/amd64
• Kubernetes version: (use kubectl version): v1.21.0
• Docker version: (use docker info): 20.10.3
• OS (e.g. from /etc/os-release): Alpine Linux 3.13.5 with 5.10.29-0-lts kernel

[BenTheElder]

Any chance this might be same issue as issue 1288 ?

No. Please see: #1288 (comment)

It's the same symptom, but that symptom just means the node container exited before we could do anything.

Please run kind create cluster --retain (prevent cleanup, can manually clean up with kind delete cluster) and upload kind export logs output before cleaning up.

The node container crashing usually means something is wrong with the host environment (e.g. lacking cgroups)

[bdutta]

Probably I missed the point that all of the kind commands need to be executed as root. Is that true ? I am only guessing this because -- (1) I see mention about how to use kind with rootless Docker, (2) I also seemed to have missed the part that the kind create cluster was trying to create '/kind' directory and failed. Anyhow, so I deleted the kind cluster, and attempted cluster creation with sudo prefixed to all kind commands, still failed. So here is the output of sudo kind export logs:

ERROR: [command "docker exec --privileged kind-control-plane sh -c 'tar --hard-dereference -C /var/log/ -chf - . || (r=$?; [ $r -eq 1 ] || exit $r)'" failed with error: exit status 1, [command "docker exec --privileged kind-control-plane journalctl --no-pager -u containerd.service" failed with error: exit status 1, command "docker exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 1, command "docker exec --privileged kind-control-plane journalctl --no-pager" failed with error: exit status 1, command "docker exec --privileged kind-control-plane journalctl --no-pager -u kubelet.service" failed with error: exit status 1]]

[BenTheElder]

Couple things:

• it needs to be run with access to the docker socket if using docker. https://kind.sigs.k8s.io/docs/user/known-issues/#docker-permission-denied
• Also if your docker is rootless that will not work until the next release. kubernetes also officially has no support for it. https://kind.sigs.k8s.io/docs/user/rootless/
• kind create cluster only creates the kubecondig file on the host which is in $HOME/. config/kube. A /kind directory does exist inside the container but it doesn't get created by kind it gets created by docker because it is part of the image

[BenTheElder]

@bdutta I deeply suspect this is now effectively a dupe of #2236 which has more details.

even though this was filed earlier, we have more relevant details in the other issue, so I'm going to tentatively dedupe this one in favor of the other issue.

we've fixed the log exporting issue I think btw, it should work properly from a binary at head, you can download one here or build one https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64