Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error during kubeadm init - addon phase with coreDNS #2699

Closed
felipefrocha opened this issue May 25, 2022 · 16 comments
Closed

Error during kubeadm init - addon phase with coreDNS #2699

felipefrocha opened this issue May 25, 2022 · 16 comments
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@felipefrocha
Copy link

What keywords did you search in kubeadm issues before filing this one?

coredns, addons, thoubleshooting

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

kubeadm init --control-plane-endpoint=k8s-haproxy:6443 --cri-socket=unix:///var/run/containerd/containerd.sock --upload-certs --v=5
I0525 03:53:40.842427   22667 interface.go:432] Looking for default routes with IPv4 addresses
I0525 03:53:40.842449   22667 interface.go:437] Default route transits interface "enp1s0"
I0525 03:53:40.842550   22667 interface.go:209] Interface enp1s0 is up
I0525 03:53:40.842576   22667 interface.go:257] Interface "enp1s0" has 4 addresses :[x.x.x.75/25 2001:12f0:601:a94d:b333:5c87:3638:d40f/64 2001:12f0:601:a94d:8f00:de38:4b19:d702/64 fe80::1c9e:ab1d:2560:d583/64].
I0525 03:53:40.842587   22667 interface.go:224] Checking addr  x.x.x.75/25.
I0525 03:53:40.842592   22667 interface.go:231] IP found x.x.x.75
I0525 03:53:40.842597   22667 interface.go:263] Found valid IPv4 address x.x.x.75 for interface "enp1s0".
I0525 03:53:40.842602   22667 interface.go:443] Found active IP x.x.x.75 
I0525 03:53:40.842613   22667 kubelet.go:214] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
I0525 03:53:40.845410   22667 version.go:186] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
[init] Using Kubernetes version: v1.24.0
[preflight] Running pre-flight checks
I0525 03:53:42.532471   22667 checks.go:570] validating Kubernetes and kubeadm version
I0525 03:53:42.532517   22667 checks.go:170] validating if the firewall is enabled and active
I0525 03:53:42.545091   22667 checks.go:205] validating availability of port 6443
I0525 03:53:42.545189   22667 checks.go:205] validating availability of port 10259
I0525 03:53:42.545204   22667 checks.go:205] validating availability of port 10257
I0525 03:53:42.545219   22667 checks.go:282] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0525 03:53:42.545226   22667 checks.go:282] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0525 03:53:42.545231   22667 checks.go:282] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0525 03:53:42.545235   22667 checks.go:282] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0525 03:53:42.545239   22667 checks.go:432] validating if the connectivity type is via proxy or direct
I0525 03:53:42.545249   22667 checks.go:471] validating http connectivity to first IP address in the CIDR
I0525 03:53:42.545258   22667 checks.go:471] validating http connectivity to first IP address in the CIDR
I0525 03:53:42.545262   22667 checks.go:106] validating the container runtime
I0525 03:53:42.551628   22667 checks.go:331] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0525 03:53:42.551671   22667 checks.go:331] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0525 03:53:42.551694   22667 checks.go:646] validating whether swap is enabled or not
I0525 03:53:42.551716   22667 checks.go:372] validating the presence of executable crictl
I0525 03:53:42.551734   22667 checks.go:372] validating the presence of executable conntrack
I0525 03:53:42.551742   22667 checks.go:372] validating the presence of executable ip
I0525 03:53:42.551751   22667 checks.go:372] validating the presence of executable iptables
I0525 03:53:42.551761   22667 checks.go:372] validating the presence of executable mount
I0525 03:53:42.551769   22667 checks.go:372] validating the presence of executable nsenter
I0525 03:53:42.551778   22667 checks.go:372] validating the presence of executable ebtables
I0525 03:53:42.551786   22667 checks.go:372] validating the presence of executable ethtool
I0525 03:53:42.551794   22667 checks.go:372] validating the presence of executable socat
I0525 03:53:42.551803   22667 checks.go:372] validating the presence of executable tc
I0525 03:53:42.551810   22667 checks.go:372] validating the presence of executable touch
I0525 03:53:42.551819   22667 checks.go:518] running all checks
	[WARNING SystemVerification]: missing optional cgroups: blkio
I0525 03:53:42.558121   22667 checks.go:403] checking whether the given node name is valid and reachable using net.LookupHost
I0525 03:53:42.558132   22667 checks.go:612] validating kubelet version
I0525 03:53:42.596489   22667 checks.go:132] validating if the "kubelet" service is enabled and active
I0525 03:53:42.605557   22667 checks.go:205] validating availability of port 10250
I0525 03:53:42.605605   22667 checks.go:205] validating availability of port 2379
I0525 03:53:42.605626   22667 checks.go:205] validating availability of port 2380
I0525 03:53:42.605647   22667 checks.go:245] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0525 03:53:42.605749   22667 checks.go:834] using image pull policy: IfNotPresent
I0525 03:53:42.613318   22667 checks.go:843] image exists: k8s.gcr.io/kube-apiserver:v1.24.0
I0525 03:53:42.619417   22667 checks.go:843] image exists: k8s.gcr.io/kube-controller-manager:v1.24.0
I0525 03:53:42.625903   22667 checks.go:843] image exists: k8s.gcr.io/kube-scheduler:v1.24.0
I0525 03:53:42.632075   22667 checks.go:843] image exists: k8s.gcr.io/kube-proxy:v1.24.0
I0525 03:53:42.639333   22667 checks.go:843] image exists: k8s.gcr.io/pause:3.7
I0525 03:53:42.645225   22667 checks.go:843] image exists: k8s.gcr.io/etcd:3.5.3-0
I0525 03:53:42.650989   22667 checks.go:843] image exists: k8s.gcr.io/coredns/coredns:v1.8.6
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0525 03:53:42.651034   22667 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0525 03:53:42.745983   22667 certs.go:522] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-haproxy k8s-ufmg-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 x.x.x.75]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0525 03:53:42.924577   22667 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0525 03:53:42.997336   22667 certs.go:522] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0525 03:53:43.058714   22667 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0525 03:53:43.142842   22667 certs.go:522] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-ufmg-master01 localhost] and IPs [x.x.x.75 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-ufmg-master01 localhost] and IPs [x.x.x.75 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0525 03:53:43.670227   22667 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0525 03:53:43.752555   22667 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0525 03:53:43.819945   22667 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0525 03:53:44.168894   22667 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0525 03:53:44.228372   22667 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I0525 03:53:44.278241   22667 kubelet.go:65] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0525 03:53:44.516612   22667 manifests.go:99] [control-plane] getting StaticPodSpecs
I0525 03:53:44.516972   22667 certs.go:522] validating certificate period for CA certificate
I0525 03:53:44.517125   22667 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0525 03:53:44.517148   22667 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I0525 03:53:44.517161   22667 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0525 03:53:44.517174   22667 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I0525 03:53:44.517190   22667 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I0525 03:53:44.521585   22667 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0525 03:53:44.521621   22667 manifests.go:99] [control-plane] getting StaticPodSpecs
I0525 03:53:44.522013   22667 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0525 03:53:44.522038   22667 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I0525 03:53:44.522052   22667 manifests.go:125] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0525 03:53:44.522065   22667 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0525 03:53:44.522082   22667 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0525 03:53:44.522097   22667 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I0525 03:53:44.522115   22667 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I0525 03:53:44.523482   22667 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0525 03:53:44.523512   22667 manifests.go:99] [control-plane] getting StaticPodSpecs
I0525 03:53:44.523892   22667 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0525 03:53:44.524752   22667 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0525 03:53:44.525901   22667 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0525 03:53:44.525926   22667 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0525 03:53:45.529278   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 1 to https://k8s-haproxy:6443/healthz?timeout=10s
I0525 03:53:46.531165   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 2 to https://k8s-haproxy:6443/healthz?timeout=10s
I0525 03:53:47.533108   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 3 to https://k8s-haproxy:6443/healthz?timeout=10s
I0525 03:53:48.535184   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 4 to https://k8s-haproxy:6443/healthz?timeout=10s
I0525 03:53:49.536948   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 5 to https://k8s-haproxy:6443/healthz?timeout=10s
I0525 03:53:50.538124   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 6 to https://k8s-haproxy:6443/healthz?timeout=10s
I0525 03:53:51.539599   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 7 to https://k8s-haproxy:6443/healthz?timeout=10s
I0525 03:53:52.541122   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 8 to https://k8s-haproxy:6443/healthz?timeout=10s
I0525 03:53:53.542833   22667 with_retry.go:241] Got a Retry-After 1s response for attempt 9 to https://k8s-haproxy:6443/healthz?timeout=10s
[apiclient] All control plane components are healthy after 13.056677 seconds
I0525 03:53:57.584614   22667 uploadconfig.go:110] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0525 03:53:57.630862   22667 uploadconfig.go:124] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
I0525 03:53:57.743024   22667 uploadconfig.go:129] [upload-config] Preserving the CRISocket information for the control-plane node
I0525 03:53:57.743060   22667 patchnode.go:31] [patchnode] Uploading the CRI Socket information "unix:///var/run/containerd/containerd.sock" to the Node API object "k8s-ufmg-master01" as an annotation
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
a1f9bc34ce92921fc6a765b6d345de3313359b5ba71a1d897d8fa89a6ae07ed7
[mark-control-plane] Marking the node k8s-ufmg-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-ufmg-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: ecg7dx.zxkrg8zgrnf36qxp
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I0525 03:53:59.981195   22667 clusterinfo.go:47] [bootstrap-token] loading admin kubeconfig
I0525 03:53:59.982340   22667 clusterinfo.go:58] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig
I0525 03:53:59.982931   22667 clusterinfo.go:70] [bootstrap-token] creating/updating ConfigMap in kube-public namespace
I0525 03:54:00.075030   22667 clusterinfo.go:84] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0525 03:54:00.130802   22667 kubeletfinalize.go:90] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0525 03:54:00.135259   22667 kubeletfinalize.go:134] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
rpc error: code = Unknown desc = malformed header: missing HTTP content-type
unable to create a new DNS service
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createDNSService
	cmd/kubeadm/app/phases/addons/dns/dns.go:247
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createCoreDNSAddon
	cmd/kubeadm/app/phases/addons/dns/dns.go:233
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.coreDNSAddon
	cmd/kubeadm/app/phases/addons/dns/dns.go:135
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.EnsureDNSAddon
	cmd/kubeadm/app/phases/addons/dns/dns.go:94
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runCoreDNSAddon
	cmd/kubeadm/app/cmd/phases/init/addons.go:93
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	cmd/kubeadm/app/cmd/init.go:153
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
	vendor/github.com/spf13/cobra/command.go:856
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	vendor/github.com/spf13/cobra/command.go:974
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
	vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
	cmd/kubeadm/app/kubeadm.go:50
main.main
	cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1571
error execution phase addon/coredns
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	cmd/kubeadm/app/cmd/init.go:153
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
	vendor/github.com/spf13/cobra/command.go:856
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	vendor/github.com/spf13/cobra/command.go:974
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
	vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
	cmd/kubeadm/app/kubeadm.go:50
main.main
	cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1571

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:44:24Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Kubernetes version (use kubectl version): Client Version: v1.24.0 | Kustomize Version: v4.5.4

  • Cloud provider or hardware configuration: dell optiplex 3070

  • OS (e.g. from /etc/os-release): Debian GNU/Linux 11 (bullseye)

  • Kernel (e.g. uname -a):5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux

  • Container runtime (CRI) (e.g. containerd, cri-o): revision="1.4.13~ds1-1~deb11u1" version="1.4.13~ds1"

  • Container networking plugin (CNI) (e.g. Calico, Cilium):

  • Others:

What happened?

During kubeadm intialization I keep receiving error

What you expected to happen?

Initialization should happen without a problem

How to reproduce it (as minimally and precisely as possible)?

In Debian Environment run initialization with the referciated tools

Anything else we need to know?

@pacoxu
Copy link
Member

pacoxu commented May 25, 2022

I suspect it‘s a problem in the k8s-haproxy configuration.

@neolit123
Copy link
Member

/kind support

@k8s-ci-robot k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label May 25, 2022
@neolit123
Copy link
Member

Likely a problem with this https://www.haproxy.com/documentation/kubernetes/latest/configuration/ingress/ and headers like @pacoxu mentioned.

We assume we have a working HA LB guide at https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

@felipefrocha are you following the same guide or something else?

If you think something can be noted in there, please help us by sending a PR.

@felipefrocha
Copy link
Author

felipefrocha commented May 30, 2022

@neolit123, I was folloing this docs, but I didn't use the keepalive, once my only node to HAproxy was well know and set on etc/hosts on my masters nodes.

# cat /etc/hosts in all masters are the same 
xxx.xxx.xxx.xxx k8s-haproxy
$ ping k8s-haproxy
PING k8s-haproxy (xxx.xxx.xxx.xxx) 56(84) bytes of data.
64 bytes from k8s-haproxy (xxx.xxx.xxx.xxx): icmp_seq=1 ttl=64 time=0.287 ms
64 bytes from k8s-haproxy (xxx.xxx.xxx.xxx): icmp_seq=2 ttl=64 time=0.569 ms
64 bytes from k8s-haproxy (xxx.xxx.xxx.xxx): icmp_seq=3 ttl=64 time=0.684 ms
64 bytes from k8s-haproxy (xxx.xxx.xxx.xxx): icmp_seq=4 ttl=64 time=0.397 ms

The configs to HAProxy follows the documentation that you mentioned:

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0 info
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the masters
#---------------------------------------------------------------------
frontend apiserver
    bind *:6443 mss 1500 
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        # server ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check
        server k8s-master01 xxx.xxx.xxx.xxx:6443 check fall 3 rise 2
        server k8s-master02 xxx.xxx.xxx.xxx:6443 check fall 3 rise 2
        server k8s-master03 xxx.xxx.xxx.xxx:6443 check fall 3 rise 2

Besides this setup a followed the instructions found at ha k8s
kubeadm init --control-plane-endpoint=k8s-haproxy:6443 --cri-socket=unix:///var/run/containerd/containerd.sock --upload-certs

@neolit123
Copy link
Member

It's not a bug in the kubeadm code.
Sounds like something in the networking setup.

Does it happen every time?

@neolit123
Copy link
Member

neolit123 commented Jun 7, 2022

you might be able to get more responses on the support channels.
https://github.com/kubernetes/kubernetes/blob/master/SUPPORT.md

i don't see a kubeadm bug, but if one is confirmed please open the issue with more details.
thanks

@yaoxin1995
Copy link

The error does come from the haproxy. To resolve it, I skip the proxy initilization part by using kubeadm init --skip-phases=addon/kube-proxy

@din14970
Copy link

Maybe a very silly thing, but for anyone that may find this issue: I ran into this because haproxy and keepalived were not installed on the node.

@lknite
Copy link

lknite commented Oct 16, 2022

If '--skip-phases=addon/kube-proxy' is used, it does let the install complete. Give it like 40 seconds and then run

kubeadm init phase addon kube-proxy \
  --control-plane-endpoint="<ha-controlplane-loadbalancer>:6443" \
  --pod-network-cidr="<put your cidr here>"

to install the kube-proxy addon successfully. (retry if you need to wait a few more seconds) ...

On centos 9 stream I had to also copy the whole containerd default configuration, then modify the systemd line

# make a copy of the default containerd configuration
containerd config default \| sudo tee /etc/containerd/config.toml

# set to use systemd
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# adjust pause image to what's actually installed
PAUSE_IMAGE=$(kubeadm config images list \| grep pause)
sudo -E sed -i "s,sandbox_image = .*,sandbox_image = \"$PAUSE_IMAGE\",g" /etc/containerd/config.toml

# restart the containerd service
sudo systemctl enable containerd
sudo systemctl restart container

@gaetanquentin
Copy link

for me the tip to install kube-proxy later do not work on ubuntu 22.04.

install logs:

Creating pod
I1020 22:01:18.020554     588 initconfiguration.go:116] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
I1020 22:01:18.020817     588 interface.go:432] Looking for default routes with IPv4 addresses
I1020 22:01:18.020853     588 interface.go:437] Default route transits interface "eth0"
I1020 22:01:18.020970     588 interface.go:209] Interface eth0 is up
I1020 22:01:18.021097     588 interface.go:257] Interface "eth0" has 2 addresses :[172.16.99.103/24 fe80::216:3eff:fe35:6c97/64].
I1020 22:01:18.021130     588 interface.go:224] Checking addr  172.16.99.103/24.
I1020 22:01:18.021157     588 interface.go:231] IP found 172.16.99.103
I1020 22:01:18.021183     588 interface.go:263] Found valid IPv4 address 172.16.99.103 for interface "eth0".
I1020 22:01:18.021199     588 interface.go:443] Found active IP 172.16.99.103 
I1020 22:01:18.021248     588 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
I1020 22:01:18.026246     588 version.go:187] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
[init] Using Kubernetes version: v1.25.3
[preflight] Running pre-flight checks
I1020 22:01:18.395125     588 checks.go:568] validating Kubernetes and kubeadm version
I1020 22:01:18.395453     588 checks.go:168] validating if the firewall is enabled and active
I1020 22:01:18.427988     588 checks.go:203] validating availability of port 6443
I1020 22:01:18.428923     588 checks.go:203] validating availability of port 10259
I1020 22:01:18.429472     588 checks.go:203] validating availability of port 10257
I1020 22:01:18.429859     588 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I1020 22:01:18.430488     588 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I1020 22:01:18.430640     588 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I1020 22:01:18.430719     588 checks.go:280] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I1020 22:01:18.431086     588 checks.go:430] validating if the connectivity type is via proxy or direct
I1020 22:01:18.431190     588 checks.go:469] validating http connectivity to first IP address in the CIDR
I1020 22:01:18.431418     588 checks.go:469] validating http connectivity to first IP address in the CIDR
I1020 22:01:18.431588     588 checks.go:104] validating the container runtime
I1020 22:01:18.538155     588 checks.go:329] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1020 22:01:18.538308     588 checks.go:329] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1020 22:01:18.538494     588 checks.go:644] validating whether swap is enabled or not
I1020 22:01:18.539924     588 checks.go:370] validating the presence of executable crictl
I1020 22:01:18.540005     588 checks.go:370] validating the presence of executable conntrack
I1020 22:01:18.540088     588 checks.go:370] validating the presence of executable ip
I1020 22:01:18.540194     588 checks.go:370] validating the presence of executable iptables
I1020 22:01:18.540344     588 checks.go:370] validating the presence of executable mount
I1020 22:01:18.540458     588 checks.go:370] validating the presence of executable nsenter
I1020 22:01:18.540579     588 checks.go:370] validating the presence of executable ebtables
I1020 22:01:18.540772     588 checks.go:370] validating the presence of executable ethtool
I1020 22:01:18.540823     588 checks.go:370] validating the presence of executable socat
I1020 22:01:18.540939     588 checks.go:370] validating the presence of executable tc
I1020 22:01:18.541003     588 checks.go:370] validating the presence of executable touch
I1020 22:01:18.541054     588 checks.go:516] running all checks
	[WARNING SystemVerification]: missing optional cgroups: blkio
I1020 22:01:18.559476     588 checks.go:401] checking whether the given node name is valid and reachable using net.LookupHost
I1020 22:01:18.560818     588 checks.go:610] validating kubelet version
I1020 22:01:18.637591     588 checks.go:130] validating if the "kubelet" service is enabled and active
I1020 22:01:18.655474     588 checks.go:203] validating availability of port 10250
I1020 22:01:18.655752     588 checks.go:203] validating availability of port 2379
I1020 22:01:18.655954     588 checks.go:203] validating availability of port 2380
I1020 22:01:18.656140     588 checks.go:243] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I1020 22:01:18.656697     588 checks.go:832] using image pull policy: IfNotPresent
I1020 22:01:18.701153     588 checks.go:841] image exists: registry.k8s.io/kube-apiserver:v1.25.3
I1020 22:01:18.733978     588 checks.go:841] image exists: registry.k8s.io/kube-controller-manager:v1.25.3
I1020 22:01:18.763547     588 checks.go:841] image exists: registry.k8s.io/kube-scheduler:v1.25.3
I1020 22:01:18.794406     588 checks.go:841] image exists: registry.k8s.io/kube-proxy:v1.25.3
I1020 22:01:18.825161     588 checks.go:841] image exists: registry.k8s.io/pause:3.8
I1020 22:01:18.856263     588 checks.go:841] image exists: registry.k8s.io/etcd:3.5.4-0
I1020 22:01:18.885055     588 checks.go:841] image exists: registry.k8s.io/coredns/coredns:v1.9.3
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I1020 22:01:18.885191     588 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I1020 22:01:19.018908     588 certs.go:522] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kube1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.99.103]
[certs] Generating "apiserver-kubelet-client" certificate and key
I1020 22:01:19.840679     588 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I1020 22:01:20.015246     588 certs.go:522] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I1020 22:01:20.511726     588 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I1020 22:01:20.948044     588 certs.go:522] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kube1 localhost] and IPs [172.16.99.103 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kube1 localhost] and IPs [172.16.99.103 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I1020 22:01:22.099253     588 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I1020 22:01:22.225928     588 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I1020 22:01:22.605702     588 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I1020 22:01:22.992252     588 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I1020 22:01:23.284582     588 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I1020 22:01:23.466241     588 kubelet.go:66] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I1020 22:01:23.795422     588 manifests.go:99] [control-plane] getting StaticPodSpecs
I1020 22:01:23.796013     588 certs.go:522] validating certificate period for CA certificate
I1020 22:01:23.796150     588 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I1020 22:01:23.796200     588 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I1020 22:01:23.796222     588 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I1020 22:01:23.796248     588 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I1020 22:01:23.796267     588 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I1020 22:01:23.796282     588 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I1020 22:01:23.802722     588 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I1020 22:01:23.802779     588 manifests.go:99] [control-plane] getting StaticPodSpecs
I1020 22:01:23.803165     588 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I1020 22:01:23.803182     588 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I1020 22:01:23.803200     588 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I1020 22:01:23.803222     588 manifests.go:125] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I1020 22:01:23.803243     588 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I1020 22:01:23.803289     588 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I1020 22:01:23.803340     588 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I1020 22:01:23.803383     588 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I1020 22:01:23.804952     588 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I1020 22:01:23.804990     588 manifests.go:99] [control-plane] getting StaticPodSpecs
I1020 22:01:23.805399     588 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I1020 22:01:23.806468     588 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I1020 22:01:23.807771     588 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I1020 22:01:23.807813     588 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.004903 seconds
I1020 22:01:37.814914     588 uploadconfig.go:110] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I1020 22:01:37.843661     588 uploadconfig.go:124] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
I1020 22:01:37.893539     588 uploadconfig.go:129] [upload-config] Preserving the CRISocket information for the control-plane node
I1020 22:01:37.893655     588 patchnode.go:31] [patchnode] Uploading the CRI Socket information "unix:///var/run/containerd/containerd.sock" to the Node API object "kube1" as an annotation
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node kube1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node kube1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: fw1b0a.saormkli2051czi2
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I1020 22:01:39.517511     588 clusterinfo.go:47] [bootstrap-token] loading admin kubeconfig
I1020 22:01:39.522379     588 clusterinfo.go:58] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig
I1020 22:01:39.523399     588 clusterinfo.go:70] [bootstrap-token] creating/updating ConfigMap in kube-public namespace
I1020 22:01:39.615828     588 clusterinfo.go:84] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I1020 22:01:39.691676     588 kubeletfinalize.go:90] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I1020 22:01:39.697389     588 kubeletfinalize.go:134] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
[addons] Applied essential addon: CoreDNS
Post "https://172.16.99.103:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s": dial tcp 172.16.99.103:6443: connect: connection refused
unable to create RBAC clusterrolebinding
k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient.CreateOrUpdateClusterRoleBinding
	cmd/kubeadm/app/util/apiclient/idempotency.go:266
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/proxy.printOrCreateKubeProxyObjects
	cmd/kubeadm/app/phases/addons/proxy/proxy.go:139
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/proxy.EnsureProxyAddon
	cmd/kubeadm/app/phases/addons/proxy/proxy.go:63
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runKubeProxyAddon
	cmd/kubeadm/app/cmd/phases/init/addons.go:121
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	cmd/kubeadm/app/cmd/init.go:154
github.com/spf13/cobra.(*Command).execute
	vendor/github.com/spf13/cobra/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
	vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
	cmd/kubeadm/app/kubeadm.go:50
main.main
	cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594
error execution phase addon/kube-proxy
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	cmd/kubeadm/app/cmd/init.go:154
github.com/spf13/cobra.(*Command).execute
	vendor/github.com/spf13/cobra/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
	vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
	cmd/kubeadm/app/kubeadm.go:50
main.main
	cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594

some logs from journalctl:
journalctl | grep -e 'kubelet' -e 'containerd' | egrep -e '[[:space:]]+E[[:digit:]]+' -e 'failed' > ERRORS.log

Oct 20 17:04:09 kube1 containerd[1537]: time="2022-10-20T17:04:09.341290880Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exi
t status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.15.0-50-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1
Oct 20 17:04:09 kube1 containerd[1537]: time="2022-10-20T17:04:09.341890512Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Oct 20 17:04:09 kube1 containerd[1537]: time="2022-10-20T17:04:09.351655305Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config
 load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Oct 20 17:04:29 kube1 kubelet[2216]: E1020 17:04:29.949686    2216 run.go:74] "command failed" err="failed to load kubelet config file, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, err
or failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory, path: /var/lib/kubelet/config.yaml"


[..]


Oct 20 17:06:33 kube1 kubelet[2437]: E1020 17:06:33.003517    2437 run.go:74] "command failed" err="failed to load kubelet config file, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, err
or failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory, path: /var/lib/kubelet/config.yaml"
Oct 20 21:59:56 kube1 containerd[278]: time="2022-10-20T21:59:56.536168277Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit
 status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.15.0-50-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1
Oct 20 21:59:56 kube1 containerd[278]: time="2022-10-20T21:59:56.537663970Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Oct 20 21:59:56 kube1 containerd[278]: time="2022-10-20T21:59:56.553904976Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config 
load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"


[..]

Oct 20 22:01:24 kube1 kubelet[698]: E1020 22:01:24.519368     698 cri_stats_provider.go:452] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
Oct 20 22:01:24 kube1 kubelet[698]: E1020 22:01:24.519719     698 kubelet.go:1317] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Oct 20 22:01:24 kube1 kubelet[698]: W1020 22:01:24.523491     698 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSIDriver: Get "https://172.16.99.103:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 172.16.99.103:6443: connect: connection refused
Oct 20 22:01:24 kube1 kubelet[698]: E1020 22:01:24.523600     698 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://172.16.99.103:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 172.16.99.103:6443: connect: connection refused
Oct 20 22:01:24 kube1 kubelet[698]: E1020 22:01:24.524142     698 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get "https://172.16.99.103:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/kube1?timeout=10s": dial tcp 172.16.99.103:6443: connect: connection refused
Oct 20 22:01:24 kube1 kubelet[698]: E1020 22:01:24.525239     698 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Oct 20 22:01:24 kube1 kubelet[698]: E1020 22:01:24.619586     698 eviction_manager.go:256] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"kube1\" not found"
Oct 20 22:01:24 kube1 kubelet[698]: E1020 22:01:24.624177     698 kubelet.go:2448] "Error getting node" err="node \"kube1\" not found"


[..]

Oct 20 22:01:40 kube1 kubelet[1141]: E1020 22:01:40.902066    1141 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube1.171fe6060a7f2da2", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"kube1", UID:"kube1", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node kube1 status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"kube1"}, FirstTimestamp:time.Date(2022, time.October, 20, 22, 1, 40, 770672034, time.Local), LastTimestamp:time.Date(2022, time.October, 20, 22, 1, 40, 770672034, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'rpc error: code = Unknown desc = malformed header: missing HTTP content-type' (will not retry!)
 

@EthanSherr
Copy link

@gaetanquentin were you able to resolve your issue? I'm running into the same thing.

@gaetanquentin
Copy link

@gaetanquentin were you able to resolve your issue? I'm running into the same thing.

No. I had to switch to canonical microk8s.

@oculos
Copy link

oculos commented Dec 3, 2022

same problem here on ubuntu 22.04

@bestofman
Copy link

I encountered same problem on Ubuntu 22.04

@abhinand5
Copy link

If '--skip-phases=addon/kube-proxy' is used, it does let the install complete. Give it like 40 seconds and then run

kubeadm init phase addon kube-proxy \
  --control-plane-endpoint="<ha-controlplane-loadbalancer>:6443" \
  --pod-network-cidr="<put your cidr here>"

to install the kube-proxy addon successfully. (retry if you need to wait a few more seconds) ...

On centos 9 stream I had to also copy the whole containerd default configuration, then modify the systemd line

# make a copy of the default containerd configuration
containerd config default \| sudo tee /etc/containerd/config.toml

# set to use systemd
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# adjust pause image to what's actually installed
PAUSE_IMAGE=$(kubeadm config images list \| grep pause)
sudo -E sed -i "s,sandbox_image = .*,sandbox_image = \"$PAUSE_IMAGE\",g" /etc/containerd/config.toml

# restart the containerd service
sudo systemctl enable containerd
sudo systemctl restart container

It worked for me in Ubuntu 22.04 Server

Additionally I also had to clean up the Flannel CNI config files /etc/cni/net.d/*flannel* to clear previous configurations. And also clear old iptables rules.

Kubernetes: 1.24.1

@bestofman
Copy link

If '--skip-phases=addon/kube-proxy' is used, it does let the install complete. Give it like 40 seconds and then run

kubeadm init phase addon kube-proxy \
  --control-plane-endpoint="<ha-controlplane-loadbalancer>:6443" \
  --pod-network-cidr="<put your cidr here>"

to install the kube-proxy addon successfully. (retry if you need to wait a few more seconds) ...
On centos 9 stream I had to also copy the whole containerd default configuration, then modify the systemd line

# make a copy of the default containerd configuration
containerd config default \| sudo tee /etc/containerd/config.toml

# set to use systemd
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# adjust pause image to what's actually installed
PAUSE_IMAGE=$(kubeadm config images list \| grep pause)
sudo -E sed -i "s,sandbox_image = .*,sandbox_image = \"$PAUSE_IMAGE\",g" /etc/containerd/config.toml

# restart the containerd service
sudo systemctl enable containerd
sudo systemctl restart container

It worked for me in Ubuntu 22.04 Server

Additionally I also had to clean up the Flannel CNI config files /etc/cni/net.d/*flannel* to clear previous configurations. And also clear old iptables rules.

Kubernetes: 1.24.1

I did this as well and mine is working too as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Development

No branches or pull requests