[Release-1.28] - k3s crashes if I do not start the cloud-provider

[brandond]

Backport fix for k3s crashes if I do not start the cloud-provider

• k3s crashes if I do not start the cloud-provider #10068

[fmoral2]

Validated on Version:

-$   k3s version v1.28.10+k3s-7de7adb2 (7de7adb2)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu
AMD

Cluster Configuration:
-1 nodes

config;

 sudo bash -c 'cat <<EOF>>/etc/rancher/k3s/config.yaml
write-kubeconfig-mode: 644
secrets-encryption: true
cluster-init: true
selinux: true
kubelet-arg:
 - cloud-provider=external
disable-cloud-controller: true
disable: servicelb
EOF' 

Steps to validate the fix

1. start k3s with config above
2. Validate that it not crashes
3. Validate nodes and pods

Reproduction Issue:

 k3s version v1.29.5+k3s1 (4e53a323)

  journalctl -xeu k3s.service | grep "timed"
May 31 12:01:41 ip-  k3s[4434]: time="2024-05-31T12:01:41Z" level=fatal msg="network policy controller timed out waiting for node.cloudprovider.kubernetes.io/uninitialized taint to be removed from Node  : timed out waiting for the condition"

 
 :~$ systemctl status k3s  server.service
Unit server.service could not be found.
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2024-05-31 12:04:28 UTC; 5s ago
       Docs: https://k3s.io
    Process: 4934 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 4936 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 4937 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 4938 ExecStart=/usr/local/bin/k3s server --cluster-init --token secret --write-kubeconfig-mode=644 (code=exited, status=1/FAILURE)
   Main PID: 4938 (code=exited, status=1/FAILURE)
        CPU: 29.239s

Validation Results:

$ journalctl -xeu k3s.service | grep "timed"
<no logs>

  $ systemctl status k3s  server.service
Unit server.service could not be found.
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running)  
       Docs: https://k3s.io


:~$ systemctl status k3s  server.service
Unit server.service could not be found.
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running)  

 
 k get nodes,pods -A
NAME                    STATUS   ROLES                       AGE   VERSION
node/ip   Ready    control-plane,etcd,master   89s   v1.28.10+k3s-7de7adb2

NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-6799fbcd5-q92l6                  0/1     Pending   0          73s
kube-system   pod/helm-install-traefik-crd-v597m           0/1     Pending   0          73s
kube-system   pod/helm-install-traefik-vn459               0/1     Pending   0          73s
kube-system   pod/local-path-provisioner-6f5d79df6-s844t   0/1     Pending   0          73s
kube-system   pod/metrics-server-54fd9b65b-4s6gr           0/1     Pending   0          73s