Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CrashLoopBackOff right after kubeadm init #2701

Closed
umerkle opened this issue May 29, 2022 · 2 comments
Closed

CrashLoopBackOff right after kubeadm init #2701

umerkle opened this issue May 29, 2022 · 2 comments
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@umerkle
Copy link

umerkle commented May 29, 2022

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:24:38Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4

  • Cloud provider or hardware configuration:

x86_64, AMD Ryzen 5 3600 6-Core Processor, 64GB RAM, 2TB HDD

  • OS (e.g. from /etc/os-release):

Ubuntu 22.04 LTS

  • Kernel (e.g. uname -a):

5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

  • Container runtime (CRI) (e.g. containerd, cri-o):

containerd

  • Container networking plugin (CNI) (e.g. Calico, Cilium):

Calico
Currently resetted kubeadm again and not yet added Calico.

What happened?

Ran

sudo kubeadm init --config kubeadm-config.yaml 

Init failed in phase [addons].

error execution phase addon/kube-proxy: unable to create ConfigMap: rpc error: code = Unknown desc = malformed header: missing HTTP content-type

Tried to ran phase manually with

sudo kubeadm init phase addon kube-proxy

Was successful.
But kube-controller-manager, kube-proxy and kube-scheduler constantly have CrashLoopBackOff

Tried to restart kubelet. But right after restart again in CrashLoopBackOff. After some time kubelet isn't reachable and i get the following message:

The connection to the server XX.XXX.XXX.XXX:6443 was refused - did you specify the right host or port?

What you expected to happen?

Successful init and a running controller node with coredns, etcd, kube-apiserver, kubecontroller-manager, kube-proxy, kube-scheduler

How to reproduce it (as minimally and precisely as possible)?

Running

sudo kubeadm init --config kubeadm-config.yaml 

With a kubeadm-config.yaml that looks like:

# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: "/var/run/containerd/containerd.sock"
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.24.1
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

Anything else we need to know?

According to this page, there should be no CrashLoopBackOffs right after kubeadm init
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/
Tried to start from this tutorial:
https://jimangel.io/post/dockershim-kubernetes-v1.24/
Came until "Deploy a “hello-world” workload"
After that had various kublet restarts and kubeadm reset and inits

@neolit123
Copy link
Member

/kind support

missing HTTP content-type

Same report here #2699 but we suspected it was ha proxy. Make sure nothing is stripping headers in your network setup. We don't have such problems in our CI.

The crash looping can be caused by container runtime problems or network peoblems and you need to look at the kubelet and container logs.

I don't see a kubeadm bug here, per se.

@k8s-ci-robot k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label May 29, 2022
@umerkle
Copy link
Author

umerkle commented Jun 1, 2022

OK.
Did a whole new server install.
Now it seems to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Development

No branches or pull requests

3 participants