[Release-1.28] - leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur

[brandond]

Backport fix for leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur

• leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur #10046

[VestigeJ]

Reproduced here #10046 (comment) but also present in the latest release of this branch

$ kg cm -n kube-system

NAME                                                   DATA   AGE
chart-content-traefik                                  0      9m14s
chart-content-traefik-crd                              0      9m14s
cluster-dns                                            2      9m19s
coredns                                                2      9m17s
extension-apiserver-authentication                     6      9m22s
k3s-etcd-snapshot-extra-metadata                       1      3m48s
k3s-etcd-snapshots                                     4      2m24s
kube-apiserver-legacy-service-account-token-tracking   1      9m22s
kube-root-ca.crt                                       1      9m5s
local-path-config                                      4      9m17s

$ kg leases k3s -n kube-system

NAME   HOLDER             AGE
k3s    ip-1-1-1-221     9m26s

$ kg leases k3s-etcd -n kube-system

NAME       HOLDER           AGE
k3s-etcd   ip-1-1-1-5     9m31s

A few snapshots later....

Should read 14 snapshots

$ kg cm -n kube-system

NAME                                                   DATA   AGE
chart-content-traefik                                  0      10m
chart-content-traefik-crd                              0      10m
cluster-dns                                            2      10m
coredns                                                2      10m
extension-apiserver-authentication                     6      10m
k3s-etcd-snapshot-extra-metadata                       1      5m10s
k3s-etcd-snapshots                                     4      3m46s
kube-apiserver-legacy-service-account-token-tracking   1      10m
kube-root-ca.crt                                       1      10m
local-path-config                                      4      10m

##Environment Details
Reproduced using VERSION=v1.28.10+k3s1
Validated using COMMIT=a29d5552c5d60091c7e96e32fbabd2a94e3797c5

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME               STATUS   ROLES                       AGE   VERSION
ip-1-1-1-71        Ready    control-plane,etcd,master   10m   v1.28.10+k3s1
ip-1-1-1-31        Ready    <none>                      10m   v1.28.10+k3s1
ip-1-1-1-21        Ready    control-plane,etcd,master   10m   v1.28.10+k3s1
ip-1-1-1-51        Ready    control-plane,etcd,master   10m   v1.28.10+k3s1

Config.yaml:

node-external-ip: 1.1.1.71
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
cluster-init: true
embedded-registry: true

Steps

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ VERSION=v1.28.10+k3s1
$ sudo INSTALL_K3S_VERSION=$VERSION INSTALL_K3S_EXEC=server ./install-k3s.sh
$ kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=test=ing
$ kg cm -n kube-system
$ kg lease k3s -n kube-system
$ kg lease k3s-etcd -n kube-system
$ k edit lease k3s-etcd -n kube-system
$ kg lease k3s-etcd -n kube-system
$ kg cm -n kube-system
$ kg leases k3s -n kube-system
$ kg leases k3s-etcd -n kube-system
$ kg cm -n kube-system

Results:

$ kg leases k3s -n kube-system

NAME   HOLDER             AGE
k3s    ip-1-1-1-21       4m2s

$ kg leases k3s-etcd -n kube-system

NAME       HOLDER             AGE
k3s-etcd   ip-1-1-1-21       4m6s

$ k edit lease k3s-etcd -n kube-system

lease.coordination.k8s.io/k3s-etcd edited

$ kg leases k3s-etcd -n kube-system

NAME       HOLDER             AGE
k3s-etcd   ip-1-1-1-71       5m7s

$ kg cm -n kube-system

NAME                                                   DATA   AGE
chart-content-traefik                                  0      5m10s
chart-content-traefik-crd                              0      5m10s
cluster-dns                                            2      5m15s
coredns                                                2      5m13s
extension-apiserver-authentication                     6      5m18s
k3s-etcd-snapshot-extra-metadata                       1      2m5s
k3s-etcd-snapshots                                     2      94s
kube-apiserver-legacy-service-account-token-tracking   1      5m18s
kube-root-ca.crt                                       1      5m1s
local-path-config                                      4      5m13s

$ kg cm -n kube-system

NAME                                                   DATA   AGE
chart-content-traefik                                  0      5m25s
chart-content-traefik-crd                              0      5m25s
cluster-dns                                            2      5m30s
coredns                                                2      5m28s
extension-apiserver-authentication                     6      5m33s
k3s-etcd-snapshot-extra-metadata                       1      2m20s
k3s-etcd-snapshots                                     11     109s
kube-apiserver-legacy-service-account-token-tracking   1      5m33s
kube-root-ca.crt                                       1      5m16s
local-path-config                                      4      5m28s