v1.26.13+rke2r1 and v1.27.10+rke2r1 rke2 on-demand snapshot fails if hostname is in uppercase

[shindebshekhar]

Environmental Info:
RKE2 Version:
Reproduced issue on both the below rke2 versions -
v1.26.13+rke2r1
v1.27.10+rke2r1

Node(s) CPU architecture, OS, and Version:
Linux HOSTNAME 5.3.18-150300.59.150-default #1 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Tested on 3 master 3 worker node
Tested on 1 master 1 worker node

Describe the bug:

We observed that if the hostname of the master node is in upper case on-demand etcd snapshot fails with below error

rke2 etcd-snapshot save --name pre-upgrade-snapshot

HOSTNAME:~ # rke2 etcd-snapshot save --name pre-upgrade-snapshot
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --tls-san found in config.yaml, skipping
WARN[0000] Unknown flag --cni found in config.yaml, skipping
WARN[0000] Unknown flag --profile found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-cidr found in config.yaml, skipping
WARN[0000] Unknown flag --service-cidr found in config.yaml, skipping
WARN[0000] Unknown flag --disable-cloud-controller found in config.yaml, skipping
WARN[0000] Unknown flag --cloud-provider-name found in config.yaml, skipping
WARN[0000] Unknown flag --kubelet-arg found in config.yaml, skipping
WARN[0000] Unknown flag --kubelet-arg found in config.yaml, skipping
WARN[0000] Unknown flag --pod-security-admission-config-file found in config.yaml, skipping
WARN[0000] Unknown flag --node-taint found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --audit-policy-file found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-retention found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /depot/etcd_snapshots/clxxx/HOSTNAME/pre-upgrade-snapshot-HOSTNAME-1715255964
{"level":"info","ts":"2024-05-09T19:59:24.509538+0800","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/depot/etcd_snapshots/clxxx/HOSTNAME/pre-upgrade-snapshot-HOSTNAME-1715255964.part"}
{"level":"info","ts":"2024-05-09T19:59:24.509767+0800","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-05-09T19:59:24.509902+0800","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2024-05-09T19:59:44.178404+0800","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2024-05-09T19:59:44.187407+0800","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"113 MB","took":"19 seconds ago"}
{"level":"info","ts":"2024-05-09T19:59:44.192783+0800","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/depot/etcd_snapshots/clxxx/HOSTNAME/pre-upgrade-snapshot-HOSTNAME-1715255964"}
INFO[0019] Reconciling ETCDSnapshotFile resources
INFO[0020] Creating ETCDSnapshotFile for etcd-snapshot-HOSTNAME-1714633203
INFO[0020] Creating ETCDSnapshotFile for etcd-snapshot-HOSTNAME-1714849200
INFO[0020] Creating ETCDSnapshotFile for etcd-snapshot-HOSTNAME-1714964404
INFO[0020] Creating ETCDSnapshotFile for etcd-snapshot-HOSTNAME-1715097603
INFO[0020] Creating ETCDSnapshotFile for etcd-snapshot-HOSTNAME-1715180404

INFO[0201] Reconciliation of ETCDSnapshotFile resources complete
FATA[0201] nodes "HOSTNAME" not found

Expected results:

Snapshot should complete without any fatal errors.

Actual results:

INFO[0201] Reconciliation of ETCDSnapshotFile resources complete
FATA[0201] nodes "HOSTNAME" not found

[brandond]

v1.26.13+rke2r1
v1.27.10+rke2r1

These are not the most recent releases of RKE2. The rke2 etcd-snapshot save command has recently been rewritten to address multiple issues, see k3s-io/k3s#9816 for more information.

Please upgrade to the most recent release of RKE2 and test again.