[Release-1.29] - k3s crashes if I do not start the cloud-provider

[brandond]

Backport fix for k3s crashes if I do not start the cloud-provider

• k3s crashes if I do not start the cloud-provider #10068

[fmoral2]

Validated on Version:

-$  k3s version v1.29.5+k3s-485eaf31 (485eaf31)
 

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu
AMD

Cluster Configuration:
-1 nodes

config;

 sudo bash -c 'cat <<EOF>>/etc/rancher/k3s/config.yaml
write-kubeconfig-mode: 644
secrets-encryption: true
cluster-init: true
selinux: true
kubelet-arg:
 - cloud-provider=external
disable-cloud-controller: true
disable: servicelb
EOF' 

Steps to validate the fix

1. start k3s with config above
2. Validate that it not crashes
3. Validate nodes and pods

Reproduction Issue:

 k3s version v1.29.5+k3s1 (4e53a323)

  journalctl -xeu k3s.service | grep "timed"
May 31 12:01:41 ip-  k3s[4434]: time="2024-05-31T12:01:41Z" level=fatal msg="network policy controller timed out waiting for node.cloudprovider.kubernetes.io/uninitialized taint to be removed from Node  : timed out waiting for the condition"

$ ^C
 :~$ systemctl status k3s  server.service
Unit server.service could not be found.
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2024-05-31 12:04:28 UTC; 5s ago
       Docs: https://k3s.io
    Process: 4934 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 4936 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 4937 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 4938 ExecStart=/usr/local/bin/k3s server --cluster-init --token secret --write-kubeconfig-mode=644 (code=exited, status=1/FAILURE)
   Main PID: 4938 (code=exited, status=1/FAILURE)
        CPU: 29.239s

Validation Results:

$ journalctl -xeu k3s.service | grep "timed"
<no logs>

  $ systemctl status k3s  server.service
Unit server.service could not be found.
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running)  
       Docs: https://k3s.io


:~$ systemctl status k3s  server.service
Unit server.service could not be found.
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running)  


k get nodes,pods -A
NAME                    STATUS   ROLES                       AGE     VERSION
node/ip    Ready    control-plane,etcd,master   2m19s   v1.29.5+k3s-485eaf31

NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-6799fbcd5-vwjk8                  0/1     Pending   0          2m2s
kube-system   pod/helm-install-traefik-crd-tcskx           0/1     Pending   0          2m3s
kube-system   pod/helm-install-traefik-twmln               0/1     Pending   0          2m3s
kube-system   pod/local-path-provisioner-6f5d79df6-7tj4n   0/1     Pending   0          2m2s
kube-system   pod/metrics-server-54fd9b65b-pvb8j           0/1     Pending   0          2m2s