[Release-1.29] - Secrets Reencrypt command throws fatal error client timing out when there are 1000 basic secrets to reencrypt

[dereknola]

Backport fix for Secrets Reencrypt command throws fatal error client timing out when there are 1000 basic secrets to reencrypt

• Secrets Reencrypt command throws fatal error client timing out when there are 1000 basic secrets to reencrypt #11384

[aganesh-suse]

Validated on release-1.29 branch with version v1.29.12-rc1+k3s1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 etcd, 2 cp and 1 agent node.

Describe the bug:

Etcd Config.yaml:

token: xxxx
disable-apiserver: true
disable-controller-manager: true
disable-scheduler: true
node-taint:
- node-role.kubernetes.io/etcd:NoExecute
cluster-init: true
write-kubeconfig-mode: "0644"
secrets-encryption: true
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
debug: true

Control Plane Config.yaml:

token: xxxx
server: https://1.1.1.1:6443
disable-etcd: true
node-taint:
- node-role.kubernetes.io/control-plane:NoSchedule
write-kubeconfig-mode: "0644"
secrets-encryption: true
node-external-ip: 4.4.4.4
node-label:
- k3s-upgrade=server
debug: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.29.12-rc1+k3s1' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Create 1000 basic secrets:

echo 'this is a file' > file.txt && for i in {1..1000}; do echo test$i >> file.txt; kubectl create secret generic test$i --from-file=file.txt; done

5. Perform the secrets encryption operations: prepare/rotate/reencrypt (Reboot ALL server nodes after every command - primary etcd first, all other etcd next, then all cp nodes)

$ sudo k3s secrets-encrypt prepare
$ sudo k3s secrets-encrypt rotate
$ sudo k3s secrets-encrypt reencrypt

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.29.12-rc1+k3s1 (ab3818c6)
go version go1.22.9

There were no fatal errors found on running the prepare/rotate/reencrypt workflow.
Current Output of reencrypt command (that had failed previously):

$ sudo /usr/local/bin/k3s secrets-encrypt reencrypt  
time="2024-12-13T02:24:12Z" level=debug msg="Asset dir /var/lib/rancher/k3s/data/bbbbff7a3b59125bd3a40c4f1df4e21ce659e32ea9550275271bda4e395617a1"
time="2024-12-13T02:24:12Z" level=debug msg="Running /var/lib/rancher/k3s/data/bbbbff7a3b59125bd3a40c4f1df4e21ce659e32ea9550275271bda4e395617a1/bin/k3s-secrets-encrypt [/usr/local/bin/k3s secrets-encrypt reencrypt]"
reencryption started

$ journalctl -xeu k3s | grep 'SecretsProgress' 
Dec 13 02:24:13 ip-172-31-5-173 k3s[9091]: I1213 02:24:13.095812    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 50 secrets"
Dec 13 02:24:13 ip-172-31-5-173 k3s[9091]: I1213 02:24:13.821694    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 100 secrets"
Dec 13 02:24:14 ip-172-31-5-173 k3s[9091]: I1213 02:24:14.484424    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 150 secrets"
Dec 13 02:24:15 ip-172-31-5-173 k3s[9091]: I1213 02:24:15.124998    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 200 secrets"
Dec 13 02:24:15 ip-172-31-5-173 k3s[9091]: I1213 02:24:15.713813    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 250 secrets"
Dec 13 02:24:16 ip-172-31-5-173 k3s[9091]: I1213 02:24:16.487240    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 300 secrets"
Dec 13 02:24:17 ip-172-31-5-173 k3s[9091]: I1213 02:24:17.210524    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 350 secrets"
Dec 13 02:24:18 ip-172-31-5-173 k3s[9091]: I1213 02:24:18.094194    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 400 secrets"
Dec 13 02:24:18 ip-172-31-5-173 k3s[9091]: I1213 02:24:18.915739    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 450 secrets"
Dec 13 02:24:19 ip-172-31-5-173 k3s[9091]: I1213 02:24:19.650126    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 500 secrets"
Dec 13 02:24:20 ip-172-31-5-173 k3s[9091]: I1213 02:24:20.437684    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 550 secrets"
Dec 13 02:24:21 ip-172-31-5-173 k3s[9091]: I1213 02:24:21.118821    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 600 secrets"
Dec 13 02:24:21 ip-172-31-5-173 k3s[9091]: I1213 02:24:21.824015    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 650 secrets"
Dec 13 02:24:22 ip-172-31-5-173 k3s[9091]: I1213 02:24:22.512847    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 700 secrets"
Dec 13 02:24:23 ip-172-31-5-173 k3s[9091]: I1213 02:24:23.241251    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 750 secrets"
Dec 13 02:24:24 ip-172-31-5-173 k3s[9091]: I1213 02:24:24.008474    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 800 secrets"
Dec 13 02:24:24 ip-172-31-5-173 k3s[9091]: I1213 02:24:24.766515    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 850 secrets"
Dec 13 02:24:25 ip-172-31-5-173 k3s[9091]: I1213 02:24:25.662457    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 900 secrets"
Dec 13 02:24:26 ip-172-31-5-173 k3s[9091]: I1213 02:24:26.466621    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 950 secrets"
Dec 13 02:24:27 ip-172-31-5-173 k3s[9091]: I1213 02:24:27.163817    9091 event.go:376] "Event occurred" object="ip-172-31-5-173" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 1000 secrets"

Post server restart status:

$ sudo /usr/local/bin/k3s secrets-encrypt status
time="2024-12-13T02:35:32Z" level=debug msg="Asset dir /var/lib/rancher/k3s/data/bbbbff7a3b59125bd3a40c4f1df4e21ce659e32ea9550275271bda4e395617a1"
time="2024-12-13T02:35:32Z" level=debug msg="Running /var/lib/rancher/k3s/data/bbbbff7a3b59125bd3a40c4f1df4e21ce659e32ea9550275271bda4e395617a1/bin/k3s-secrets-encrypt [/usr/local/bin/k3s secrets-encrypt status]"
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2024-12-13T02:09:36Z