Installation fail: kubeadm [Initialize first master] #5139
Labels
kind/bug
Categorizes issue or PR as related to a bug.
Comments
|
Do your nodes already have kubernetes services? The ports are already in use. |
|
same problem on openstack |
|
I received this kind of error yesterday and I understood the error can't be one of the warnings you can find in the logs, since it's only warnings and mostly because the error is thrown after the timeout is reached (and 3 retries). In my case, I had an issue with the This should give you some hints. |
|
Hi, Had the same issue. In my case it was a IP<>Address problem in kubernetes.tld should match 10.1.10.127. So either update the DNS record or change the variable. Hope it will help someone ! |
|
any updates? have the same issue with cloud_provider: aws, the same scenario ( |
|
same problem on openstack. I've tried the above but the install does not finish the TASK [kubernetes/master : kubeadm | Initialize first master] . Any help would be appreciated. |
try to add |
doesn't help in case AWS, still ports in use and [kubernetes/master : kubeadm | Initialize first master] failed |
Still the same issue on OpenStack. |
|
@rstriedl5c you know, i faced with lot of problem with kubespray+terraform+openstack. in future you will face with problems in openstack like - insufficient rules in SG groups and so on... |
|
@ppcololo Thanks for the information. What's odd it can't talk the api endpoint on 6443 on my master. I've open the world to the security group. see logs below. I'm trying to use flannel vs calico to start as the cni. Also you can see my /etc/hosts file is updated with private ip's but not the floating ip's. |
|
do you use flannel cni? and then see journal and |
|
@ppcololo I did not have this file, I've added to my master. I don't see any change to my journal or kubelet status. Do I need to run the ansible playbook? |
|
what network plugin do you use? flannel/calico/canal? |
|
@ppcololo I'm using flannel. I did the restart of kubelet below. |
|
@rstriedl5c kubelet running |
|
Guys, look on this In my case it helps. |
|
@olehhordiienko Thanks. We are both using OpenStack. The fix appears to be for AWS. |
|
@rstriedl5c all changes i made: all.yml
now I have well installed k8s cluster in openstack. but you know - i tried another tool (kops) and got cluster without a lot of pain (they added openstack support with LB) |
|
@ppcololo I believe you're using CentOS, correct? here are my configs since, I'm using Ubuntu. in my k8s-cluster.yml: I've added the SG rules in compute module. I've used Kops on AWS, and other clouds before. Can you send me the Kops command you ran to create the OpenStack cluster? Plus any other things to consider with Kops OpenStack K8s cluster. Thanks in advance. I will try the above Kubespray changes and let you know if it works for me. Thanks again. |
|
@ppcololo silly question, how you generating your inventory.ini file? Does your's look similar to the following....I'm trying to setup GlusterFS nodes too. At this time, not using a bastion host. |
|
@rstriedl5c kubespray uses python script which parse terraform.tfstate file - this is inventory for ansible, i dont have another. |
|
hello, i got very similar issue with kubespray with first master results every time error with the following error. i have the hosts on DNS and i can ping . also kubelet service is error due to files missing (/etc/kubernetes/ssl/ca.crt:) fatal: [lvpaldbsvm28]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["timeout", "-k", "300s", "300s", "/usr/local/bin/kubeadm", "init", "--config=/etc/kubernetes/kubeadm-config.yaml", "--ignore-preflight-errors=all", "--skip-phases=addon/coredns", "--upload-certs"], "delta": "0:00:00.108208", "end": "2019-10-20 15:59:24.267983", "failed_when_result": true, "msg": "non-zero return code", "rc": 3, "start": "2019-10-20 15:59:24.159775", "stderr": "[apiServer.certSANs: Invalid value: "lvpaldbsvm28.pal.sap.corp\u00a0": altname is not a valid IP address, DNS label or a DNS label kubelet service shows the following error.erver.go:251] unable to load client CA file /etc/kubernetes/ssl/ca.crt: open /etc/kubernetes/ssl/ca.crt: no such ... or directory |
|
Were you able to resolve the error that caused Failure on Please let us know, it might help me with my issue. Thanks |
|
Hit with the same but only on my second and third test install. I'll see if i can find what config option i'm using which triggers it. |
|
any news here? |
You have a similar problem? |
|
Yeah, I've found that if I disable loadbalancer_apiserver option setup is going successfully. I use the latest release of kubespray from github and kubernetes 1.16.6 |
|
I'm really sorry, but in this case it was my own issue, I've firewalled balancer host half an year ago and forgot about it, master hosts couldn't initialize b/c they didn't have connect to balancer. |
|
Close it since this is old issue |
I believe kubespray scripts ought to make sure that all hosts (including the load balancer where an external one is being used) are reachable. In my case, I getting this same error using KVM VM's and in my case, the load balancer VM was stuck during boot waiting to do a fsck (or ctrl+D) in order to complete boot up. |
|
worked for me |
|
facing the same issue on openstack. Trying to install kubernetes with kubespray with octavia external load balancer. |
|
Fresh install of 6 vms at ubuntu 20.04, after failing because of a missing "/etc/kubernetes" on the control plane nodes, I get hung at failure to "Initialize first master": |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment:
Cloud provider or hardware configuration:
AWS
OS (
printf "$(uname -srm)\n$(cat /etc/os-release)\n"):coreos
Version of Ansible (
ansible --version):ansible 2.7.8
Kubespray version (commit) (
git rev-parse --short HEAD):git checkout release-2.10
git checkout release-2.11
Network plugin used:
cilium 1.3.7
Copy of your inventory file:
inventory.zip
Command used to invoke ansible:
Output of ansible run:
Error
TASK [kubernetes/master : kubeadm | Initialize first master] ***************************************************************************************************************
Tuesday 03 September 2019 07:14:25 +0000 (0:00:00.520) 0:22:02.910 *****
FAILED - RETRYING: kubeadm | Initialize first master (3 retries left).
FAILED - RETRYING: kubeadm | Initialize first master (2 retries left).
FAILED - RETRYING: kubeadm | Initialize first master (1 retries left).
fatal: [kubernetes-dev0210john0903-master0]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["timeout", "-k", "600s", "600s", "/opt/bin/kubeadm", "init", "--config=/etc/kubernetes/kubeadm-config.yaml", "--ignore-preflight-errors=all", "--skip-phases=addon/coredns", "--experimental-upload-certs", "--certificate-key=ecabe44f2d9ce1b2edbb702c8a9c77d5c84bb9cb4da05eb42fcba3dfe4ec5b5e"], "delta": "0:02:02.449063", "end": "2019-09-03 07:23:13.971380", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2019-09-03 07:21:11.522317", "stderr": "\t[WARNING Port-6443]: Port 6443 is in use\n\t[WARNING Port-10251]: Port 10251 is in use\n\t[WARNING Port-10252]: Port 10252 is in use\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists\n\t[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/\n\t[WARNING Port-10250]: Port 10250 is in use\nerror execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition", "stderr_lines": ["\t[WARNING Port-6443]: Port 6443 is in use", "\t[WARNING Port-10251]: Port 10251 is in use", "\t[WARNING Port-10252]: Port 10252 is in use", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists", "\t[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/", "\t[WARNING Port-10250]: Port 10250 is in use", "error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition"], "stdout": "[init] Using Kubernetes version: v1.14.6\n[preflight] Running pre-flight checks\n[preflight] Pulling images required for setting up a Kubernetes cluster\n[preflight] This might take a minute or two, depending on the speed of your internet connection\n[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'\n[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"\n[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"\n[kubelet-start] Activating the kubelet service\n[certs] Using certificateDir folder "/etc/kubernetes/ssl"\n[certs] Using existing ca certificate authority\n[certs] Using existing apiserver certificate and key on disk\n[certs] Using existing apiserver-kubelet-client certificate and key on disk\n[certs] Using existing front-proxy-ca certificate authority\n[certs] Using existing front-proxy-client certificate and key on disk\n[certs] External etcd mode: Skipping etcd/ca certificate authority generation\n[certs] External etcd mode: Skipping etcd/healthcheck-client certificate authority generation\n[certs] External etcd mode: Skipping etcd/server certificate authority generation\n[certs] External etcd mode: Skipping etcd/peer certificate authority generation\n[certs] External etcd mode: Skipping apiserver-etcd-client certificate authority generation\n[certs] Using the existing "sa" key\n[kubeconfig] Using kubeconfig folder "/etc/kubernetes"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"\n[control-plane] Using manifest folder "/etc/kubernetes/manifests"\n[control-plane] Creating static Pod manifest for "kube-apiserver"\n[controlplane] Adding extra host path mount "cloud-config" to "kube-apiserver"\n[controlplane] Adding extra host path mount "usr-share-ca-certificates" to "kube-apiserver"\n[controlplane] Adding extra host path mount "cloud-config" to "kube-controller-manager"\n[control-plane] Creating static Pod manifest for "kube-controller-manager"\n[controlplane] Adding extra host path mount "cloud-config" to "kube-apiserver"\n[controlplane] Adding extra host path mount "usr-share-ca-certificates" to "kube-apiserver"\n[controlplane] Adding extra host path mount "cloud-config" to "kube-controller-manager"\n[control-plane] Creating static Pod manifest for "kube-scheduler"\n[controlplane] Adding extra host path mount "cloud-config" to "kube-apiserver"\n[controlplane] Adding extra host path mount "usr-share-ca-certificates" to "kube-apiserver"\n[controlplane] Adding extra host path mount "cloud-config" to "kube-controller-manager"\n[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 5m0s\n[apiclient] All control plane components are healthy after 0.010987 seconds\n[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace\n[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster\n[kubelet-check] Initial timeout of 40s passed.", "stdout_lines": ["[init] Using Kubernetes version: v1.14.6", "[preflight] Running pre-flight checks", "[preflight] Pulling images required for setting up a Kubernetes cluster", "[preflight] This might take a minute or two, depending on the speed of your internet connection", "[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'", "[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"", "[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"", "[kubelet-start] Activating the kubelet service", "[certs] Using certificateDir folder "/etc/kubernetes/ssl"", "[certs] Using existing ca certificate authority", "[certs] Using existing apiserver certificate and key on disk", "[certs] Using existing apiserver-kubelet-client certificate and key on disk", "[certs] Using existing front-proxy-ca certificate authority", "[certs] Using existing front-proxy-client certificate and key on disk", "[certs] External etcd mode: Skipping etcd/ca certificate authority generation", "[certs] External etcd mode: Skipping etcd/healthcheck-client certificate authority generation", "[certs] External etcd mode: Skipping etcd/server certificate authority generation", "[certs] External etcd mode: Skipping etcd/peer certificate authority generation", "[certs] External etcd mode: Skipping apiserver-etcd-client certificate authority generation", "[certs] Using the existing "sa" key", "[kubeconfig] Using kubeconfig folder "/etc/kubernetes"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"", "[control-plane] Using manifest folder "/etc/kubernetes/manifests"", "[control-plane] Creating static Pod manifest for "kube-apiserver"", "[controlplane] Adding extra host path mount "cloud-config" to "kube-apiserver"", "[controlplane] Adding extra host path mount "usr-share-ca-certificates" to "kube-apiserver"", "[controlplane] Adding extra host path mount "cloud-config" to "kube-controller-manager"", "[control-plane] Creating static Pod manifest for "kube-controller-manager"", "[controlplane] Adding extra host path mount "cloud-config" to "kube-apiserver"", "[controlplane] Adding extra host path mount "usr-share-ca-certificates" to "kube-apiserver"", "[controlplane] Adding extra host path mount "cloud-config" to "kube-controller-manager"", "[control-plane] Creating static Pod manifest for "kube-scheduler"", "[controlplane] Adding extra host path mount "cloud-config" to "kube-apiserver"", "[controlplane] Adding extra host path mount "usr-share-ca-certificates" to "kube-apiserver"", "[controlplane] Adding extra host path mount "cloud-config" to "kube-controller-manager"", "[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 5m0s", "[apiclient] All control plane components are healthy after 0.010987 seconds", "[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace", "[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster", "[kubelet-check] Initial timeout of 40s passed."]}
Anything else do we need to know:
Raise both on:
release-2.10
release-2.11
Previously, seems runs correctly for some version, but now always fail.
Is below the root reason?
WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd".
The text was updated successfully, but these errors were encountered: