[Release-1.27] - Emit events for certificates about to expire

[brandond]

Backport fix for Emit events for certificates about to expire

• Emit events for certificates about to expire #9742

[ShylajaDevadiga]

Validated using latest commit id 2d48b19 on release-1.27 branch

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:
3 server 1 agent node

Config.yaml:

cat /etc/rancher/k3s/config,yaml
write-kubeconfig-mode: "0644"
tls-san:
  - fake.fqdn.value
cluster-init: true
secrets-encryption: true

Steps to reproduce the issue and validate the fix

1. Copy config.yaml
2. Set env variable CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=30
3. Install k3s
4. Check the warning when the certs are within 90 days of expiring

On server

ec2-user@ip-172-31-1-234:~> sudo /usr/local/bin/k3s certificate check
INFO[0000] Server detected, checking agent and server certificates 
INFO[0000] Checking certificates for supervisor         
WARN[0000] /var/lib/rancher/k3s/server/tls/client-supervisor.crt: certificate CN=system:k3s-supervisor,O=system:masters will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-supervisor.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for kubelet            
WARN[0000] /var/lib/rancher/k3s/agent/client-kubelet.crt: certificate CN=system:node:ip-172-31-1-234,O=system:nodes will expire within 90 days at 2024-05-15T18:18:57Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-kubelet.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
WARN[0000] /var/lib/rancher/k3s/agent/serving-kubelet.crt: certificate CN=ip-172-31-1-234 will expire within 90 days at 2024-05-15T18:18:56Z 
INFO[0000] /var/lib/rancher/k3s/agent/serving-kubelet.crt: certificate CN=k3s-server-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for api-server         
WARN[0000] /var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt: certificate CN=system:apiserver,O=system:masters will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
WARN[0000] /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt: certificate CN=kube-apiserver will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt: certificate CN=k3s-server-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for admin              
WARN[0000] /var/lib/rancher/k3s/server/tls/client-admin.crt: certificate CN=system:admin,O=system:masters will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-admin.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for auth-proxy         
WARN[0000] /var/lib/rancher/k3s/server/tls/client-auth-proxy.crt: certificate CN=system:auth-proxy will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-auth-proxy.crt: certificate CN=k3s-request-header-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for cloud-controller   
WARN[0000] /var/lib/rancher/k3s/server/tls/client-k3s-cloud-controller.crt: certificate CN=k3s-cloud-controller-manager will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-k3s-cloud-controller.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for controller-manager 
WARN[0000] /var/lib/rancher/k3s/server/tls/client-controller.crt: certificate CN=system:kube-controller-manager will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-controller.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for etcd               
WARN[0000] /var/lib/rancher/k3s/server/tls/etcd/client.crt: certificate CN=etcd-client will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/etcd/client.crt: certificate CN=etcd-server-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
WARN[0000] /var/lib/rancher/k3s/server/tls/etcd/server-client.crt: certificate CN=etcd-server will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/etcd/server-client.crt: certificate CN=etcd-server-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
WARN[0000] /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt: certificate CN=etcd-peer will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt: certificate CN=etcd-peer-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for k3s-controller     
WARN[0000] /var/lib/rancher/k3s/server/tls/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-k3s-controller.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
WARN[0000] /var/lib/rancher/k3s/agent/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-k3s-controller.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for scheduler          
WARN[0000] /var/lib/rancher/k3s/server/tls/client-scheduler.crt: certificate CN=system:kube-scheduler will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-scheduler.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for kube-proxy         
WARN[0000] /var/lib/rancher/k3s/server/tls/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/server/tls/client-kube-proxy.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
WARN[0000] /var/lib/rancher/k3s/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-kube-proxy.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 

ec2-user@ip-172-31-1-234:~> kubectl get --raw /api/v1/nodes/ip-172-31-1-234/proxy/metrics | grep k3s_certificate_expiration
# HELP k3s_certificate_expiration_seconds Remaining lifetime on the certificate.
# TYPE k3s_certificate_expiration_seconds gauge
k3s_certificate_expiration_seconds{subject="CN=etcd-client",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=etcd-peer",usages="ServerAuth,ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=etcd-peer-ca@1713205134",usages="CertSign"} 3.1535999412992096e+08
k3s_certificate_expiration_seconds{subject="CN=etcd-server",usages="ServerAuth,ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=etcd-server-ca@1713205134",usages="CertSign"} 3.1535999412992096e+08
k3s_certificate_expiration_seconds{subject="CN=ip-172-31-1-234",usages="ServerAuth"} 2.591996131702952e+06
k3s_certificate_expiration_seconds{subject="CN=k3s-client-ca@1713205134",usages="CertSign"} 3.1535999412992096e+08
k3s_certificate_expiration_seconds{subject="CN=k3s-cloud-controller-manager",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=k3s-request-header-ca@1713205134",usages="CertSign"} 3.1535999412992096e+08
k3s_certificate_expiration_seconds{subject="CN=k3s-server-ca@1713205134",usages="CertSign"} 3.1535999412992096e+08
k3s_certificate_expiration_seconds{subject="CN=kube-apiserver",usages="ServerAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:admin,O=system:masters",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:apiserver,O=system:masters",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:auth-proxy",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:k3s-controller",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:k3s-supervisor,O=system:masters",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:kube-controller-manager",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:kube-proxy",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:kube-scheduler",usages="ClientAuth"} 2.591994131702952e+06
k3s_certificate_expiration_seconds{subject="CN=system:node:ip-172-31-1-234,O=system:nodes",usages="ClientAuth"} 2.591997131702952e+06
ec2-user@ip-172-31-1-234:~> 

ec2-user@ip-172-31-1-234:~> kubectl get event|grep cert
26s         Warning   CertificateExpirationWarning     node/ip-172-31-1-234   Node certificates require attention - restart k3s on this node to trigger automatic rotation: kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T18:18:54Z, kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T18:18:54Z, api-server/client-kube-apiserver.crt: certificate CN=system:apiserver,O=system:masters will expire within 90 days at 2024-05-15T18:18:54Z, api-server/serving-kube-apiserver.crt: certificate CN=kube-apiserver will expire within 90 days at 2024-05-15T18:18:54Z, admin/client-admin.crt: certificate CN=system:admin,O=system:masters will expire within 90 days at 2024-05-15T18:18:54Z, auth-proxy/client-auth-proxy.crt: certificate CN=system:auth-proxy will expire within 90 days at 2024-05-15T18:18:54Z, controller-manager/client-controller.crt: certificate CN=system:kube-controller-manager will expire within 90 days at 2024-05-15T18:18:54Z, scheduler/client-scheduler.crt: certificate CN=system:kube-scheduler will expire within 90 days at 2024-05-15T18:18:54Z, supervisor/client-supervisor.crt: certificate CN=system:k3s-supervisor,O=system:masters will expire within 90 days at 2024-05-15T18:18:54Z, cloud-controller/client-k3s-cloud-controller.crt: certificate CN=k3s-cloud-controller-manager will expire within 90 days at 2024-05-15T18:18:54Z, etcd/client.crt: certificate CN=etcd-client will expire within 90 days at 2024-05-15T18:18:54Z, etcd/server-client.crt: certificate CN=etcd-server will expire within 90 days at 2024-05-15T18:18:54Z, etcd/peer-server-client.crt: certificate CN=etcd-peer will expire within 90 days at 2024-05-15T18:18:54Z, kubelet/client-kubelet.crt: certificate CN=system:node:ip-172-31-1-234,O=system:nodes will expire within 90 days at 2024-05-15T18:18:57Z, kubelet/serving-kubelet.crt: certificate CN=ip-172-31-1-234 will expire within 90 days at 2024-05-15T18:18:56Z, k3s-controller/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-15T18:18:54Z, k3s-controller/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-15T18:18:54Z

On agent

ec2-user@ip-172-31-13-122:~> sudo /usr/local/bin/k3s certificate check
INFO[0000] Agent detected, checking agent certificates  
INFO[0000] Checking certificates for kube-proxy         
WARN[0000] /var/lib/rancher/k3s/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-kube-proxy.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for kubelet            
WARN[0000] /var/lib/rancher/k3s/agent/client-kubelet.crt: certificate CN=system:node:ip-172-31-13-122,O=system:nodes will expire within 90 days at 2024-05-15T18:21:20Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-kubelet.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
WARN[0000] /var/lib/rancher/k3s/agent/serving-kubelet.crt: certificate CN=ip-172-31-13-122 will expire within 90 days at 2024-05-15T18:21:20Z 
INFO[0000] /var/lib/rancher/k3s/agent/serving-kubelet.crt: certificate CN=k3s-server-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
INFO[0000] Checking certificates for k3s-controller     
WARN[0000] /var/lib/rancher/k3s/agent/client-k3s-controller.crt: certificate CN=system:k3s-controller will expire within 90 days at 2024-05-15T18:18:54Z 
INFO[0000] /var/lib/rancher/k3s/agent/client-k3s-controller.crt: certificate CN=k3s-client-ca@1713205134 is ok, expires at 2034-04-13T18:18:54Z 
ec2-user@ip-172-31-13-122:~>