Better detection and auto-debugging of "waiting for control plane" issues

[pipejakob]

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

TL;DR

A lot of common things can go wrong when kubeadm init gets stuck waiting for the control plane to become healthy. Instead of sitting idly, kubeadm should be proactively looking for common failures and reporting on them instead of requiring the user to manually investigate other sources of error messages.

More context

A lot of questions and issues we see around kubeadm are around a very opaque area of kubeadm init's execution, when it is simply waiting for the control plane to become ready. One such example is here, but this is a recurring theme.

The problem is that kubeadm init does a lot of initial preflight checks (which can optionally be disabled), then writes out static manifest files for the control plane, and then simply waits for the control plane to start up and become healthy. This is an asynchronous operation that requires that the kubelet be running and healthy, that kubelet be configured to find the static manifest files where kubeadm wrote them to, that kubelet actually tries to start the pods, that Docker is able to pull the images for the pods, that the pods are able to start up with the flags they've been provided, and that no other runtime errors are keeping the pods from coming up healthy. These are a lot of asynchronous steps that can fail independently, leaving the root cause error messages scattered around in different places. All the while, kubeadm init simply sits idly and waits for the full orchestration to complete successfully.

If anything does go wrong, all the user sees is that kubeadm init gets stuck waiting for the control plane to become healthy. In order to diagnose the cause, there are pretty standard investigation steps we consistently ask users to follow: make sure that the kubelet is running, check the kubelet logs, check the Docker logs for control plane components, etc.

I think a lot of these checks should be implemented by kubeadm, running in a concurrent go routine while it's still waiting for the control plane to become healthy. Just like the preflight checks, we can start simple and only add a few automated checks that we think would give the biggest return on investment, and keep adding more to cover up holes that users discover along the way.

@kubernetes/sig-cluster-lifecycle-feature-requests

[luxas]

I've also thought about this and have it in my backlog -- thanks for filing an issue.
What makes this hard is that kubeadm CLI is agnostic to how you run your kubelet.
So we can't really check logs of kubelet nor docker (you can use other CRI runtimes)

What we definitely should do is to rephrase the message to something like:
[apiclient] Waiting for the kubelet to start the Control Plane components from manifest directory "/etc/kubernetes/manifests"

As part of the upgrades thing; I'm reworking the impl of our waiting logic and adding timeouts (separate PR coming soon)
We should definitely add the checking of ps aux | grep kubelet (in golang obvs) in a concurrent goroutine and error out faster if it doesn't start.

We could also work with sig-node to improve the system verification checks to handle more cases, etc.
We could poll the kubelet API though; which will give us the most of the information (running pods, logs, etc.)

I'm adding this to v1.8, do you want implement this @pipejakob ?