Emit events for certificates about to expire

[brandond]

We should emit events attached to the Node resource when there are certs that are within 90 days of expiring. This would prompt users to restart the service to trigger certificate renewal, before the certificates are expired and things stop working.

[ShylajaDevadiga]

Validated using latest commit id a064ae2 on master branch

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:
3 server 1 agent node

Config.yaml:

cat /etc/rancher/k3s/config,yaml
write-kubeconfig-mode: "0644"
tls-san:
  - fake.fqdn.value
cluster-init: true

Steps to reproduce the issue and validate the fix

1. Copy config.yaml
2. Set env variable CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=30
3. Install k3s
4. Check the warning when the certs are within 90 days of expiring

> sudo /usr/local/bin/k3s certificate check
INFO[0000] Server detected, checking agent and server certificates 
INFO[0000] Checking certificates for admin              
WARN[0000] /var/lib/rancher/k3s/server/tls/client-admin.crt: certificate CN=system:admin,O=system:masters will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-admin.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for auth-proxy         
WARN[0000] /var/lib/rancher/k3s/server/tls/client-auth-proxy.crt: certificate CN=system:auth-proxy will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-auth-proxy.crt: certificate CN=k3s-request-header-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for scheduler          
WARN[0000] /var/lib/rancher/k3s/server/tls/client-scheduler.crt: certificate CN=system:kube-scheduler will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-scheduler.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for kube-proxy         
WARN[0000] /var/lib/rancher/k3s/server/tls/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-kube-proxy.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
WARN[0000] /var/lib/rancher/k3s/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-kube-proxy.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for kubelet            
WARN[0000] /var/lib/rancher/k3s/agent/client-kubelet.crt: certificate CN=system:node:ip-172-31-14-48,O=system:nodes will expire within 90 days at 2024-05-09T18:51:02Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-kubelet.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
WARN[0000] /var/lib/rancher/k3s/agent/serving-kubelet.crt: certificate CN=ip-172-31-14-48 will expire within 90 days at 2024-05-09T18:51:01Z 
INFO[0000] /var/lib/rancher/k3s/agent/serving-kubelet.crt: certificate CN=k3s-server-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for api-server         
WARN[0000] /var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt: certificate CN=system:apiserver,O=system:masters will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
WARN[0000] /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt: certificate CN=kube-apiserver will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt: certificate CN=k3s-server-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for cloud-controller   
WARN[0000] /var/lib/rancher/k3s/server/tls/client-k3s-cloud-controller.crt: certificate CN=k3s-cloud-controller-manager will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-k3s-cloud-controller.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for controller-manager 
WARN[0000] /var/lib/rancher/k3s/server/tls/client-controller.crt: certificate CN=system:kube-controller-manager will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-controller.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for etcd               
WARN[0000] /var/lib/rancher/k3s/server/tls/etcd/client.crt: certificate CN=etcd-client will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/etcd/client.crt: certificate CN=etcd-server-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
WARN[0000] /var/lib/rancher/k3s/server/tls/etcd/server-client.crt: certificate CN=etcd-server will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/etcd/server-client.crt: certificate CN=etcd-server-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
WARN[0000] /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt: certificate CN=etcd-peer will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt: certificate CN=etcd-peer-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for supervisor         
WARN[0000] /var/lib/rancher/k3s/server/tls/client-supervisor.crt: certificate CN=system:k3s-supervisor,O=system:masters will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-supervisor.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for k3s-controller     
WARN[0000] /var/lib/rancher/k3s/server/tls/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-k3s-controller.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
WARN[0000] /var/lib/rancher/k3s/agent/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-k3s-controller.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z  

On agent

> sudo /usr/local/bin/k3s certificate check
INFO[0000] Agent detected, checking agent certificates  
INFO[0000] Checking certificates for kube-proxy         
WARN[0000] /var/lib/rancher/k3s/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-kube-proxy.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for kubelet            
WARN[0000] /var/lib/rancher/k3s/agent/client-kubelet.crt: certificate CN=system:node:ip-172-31-6-83,O=system:nodes will expire within 90 days at 2024-05-09T18:51:12Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-kubelet.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
WARN[0000] /var/lib/rancher/k3s/agent/serving-kubelet.crt: certificate CN=ip-172-31-6-83 will expire within 90 days at 2024-05-09T18:51:11Z 
INFO[0000] /var/lib/rancher/k3s/agent/serving-kubelet.crt: certificate CN=k3s-server-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 
INFO[0000] Checking certificates for k3s-controller     
WARN[0000] /var/lib/rancher/k3s/agent/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-09T18:50:59Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-k3s-controller.crt: certificate CN=k3s-client-ca@1712688659 is ok, expires at 2034-04-07T18:50:59Z 

Metrics and Event

> kubectl get --raw /api/v1/nodes/ip-172-31-14-48/proxy/metrics | grep k3s_certificate_expiration
# HELP k3s_certificate_expiration_seconds Remaining lifetime on the certificate.
# TYPE k3s_certificate_expiration_seconds gauge
k3s_certificate_expiration_seconds{subject="CN=etcd-client",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=etcd-peer",usages="ServerAuth,ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=etcd-peer-ca@1712688659",usages="CertSign"} 3.1535999426205605e+08
k3s_certificate_expiration_seconds{subject="CN=etcd-server",usages="ServerAuth,ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=etcd-server-ca@1712688659",usages="CertSign"} 3.1535999426205605e+08
k3s_certificate_expiration_seconds{subject="CN=ip-172-31-14-48",usages="ServerAuth"} 2.591996263493534e+06
k3s_certificate_expiration_seconds{subject="CN=k3s-client-ca@1712688659",usages="CertSign"} 3.1535999426205605e+08
k3s_certificate_expiration_seconds{subject="CN=k3s-cloud-controller-manager",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=k3s-request-header-ca@1712688659",usages="CertSign"} 3.1535999426205605e+08
k3s_certificate_expiration_seconds{subject="CN=k3s-server-ca@1712688659",usages="CertSign"} 3.1535999426205605e+08
k3s_certificate_expiration_seconds{subject="CN=kube-apiserver",usages="ServerAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:admin,O=system:masters",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:apiserver,O=system:masters",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:auth-proxy",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:k3s-controller",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:k3s-supervisor,O=system:masters",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:kube-controller-manager",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:kube-proxy",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:kube-scheduler",usages="ClientAuth"} 2.591994263493534e+06
k3s_certificate_expiration_seconds{subject="CN=system:node:ip-172-31-14-48,O=system:nodes",usages="ClientAuth"} 2.591997263493534e+06
ec2-user@ip-172-31-14-48:~> 

> kubectl get event|grep cert
24m         Warning   CertificateExpirationWarning     node/ip-172-31-6-83    Node certificates require attention - restart k3s on this node to trigger automatic rotation: kubelet/client-kubelet.crt: certificate CN=system:node:ip-172-31-6-83,O=system:nodes will expire within 90 days at 2024-05-09T18:51:12Z, kubelet/serving-kubelet.crt: certificate CN=ip-172-31-6-83 will expire within 90 days at 2024-05-09T18:51:11Z, k3s-controller/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-09T18:50:59Z, kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-09T18:50:59Z