[Release-1.28] - Expose RKE2 metrics via supervisor port

[brandond]

Backport fix for Expose RKE2 metrics via supervisor port

• Expose RKE2 metrics via supervisor port #5786

[brandond]

agents are listening on the wrong port, see #5786 (comment)

[brandond]

this is breaking the embedded registry mirror on agents too... so it'll need to be fixed this release.

[aganesh-suse]

Validated on release-1.28 branch with commit b200ce9

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA : 3 server / 1 agent

Config.yaml:

Server Node Configs:

token: xxxx
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
enable-pprof: true
supervisor-metrics: true

Agent node configs:

token: xxxx
node-external-ip: 1.1.1.2
server: https://<server ip>:9345
enable-pprof: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

2. Install RKE2

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT='b200ce9189c3da65e79acad4fd0755e25905a239' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -

3. Start the RKE2 service

$ sudo systemctl enable --now rke2-server
or 
$ sudo systemctl enable --now rke2-agent

4. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

5. Verify with server and agent IPs for the curl urls on a server node:

curl -vks --cert /var/lib/rancher/k3s/server/tls/client-admin.crt --key /var/lib/rancher/k3s/server/tls/client-admin.key https://<server|agent ip>:6443/metrics
curl -vks --cert /var/lib/rancher/k3s/server/tls/client-admin.crt --key /var/lib/rancher/k3s/server/tls/client-admin.key https://<server|agent ip>:6443/debug/pprof

Validation Results:

• rke2 version used for validation:

$ rke2 -v
rke2 version v1.28.11-rc3+rke2r1 (b200ce9189c3da65e79acad4fd0755e25905a239)
go version go1.21.11 X:boringcrypto

Verified that querying all server/agent nodes works (Not providing output due to large size):

sudo curl -vks --cert /var/lib/rancher/rke2/server/tls/client-admin.crt --key /var/lib/rancher/rke2/server/tls/client-admin.key https://<server_node_ip>:9345/debug/pprof/ 

sudo curl -vks --cert /var/lib/rancher/rke2/server/tls/client-admin.crt --key /var/lib/rancher/rke2/server/tls/client-admin.key https://<server_node_ip>:9345/metrics