[release-1.28] Fix race condition panic in loadbalancer.nextServer

[brandond]

Proposed Changes

Fix race condition panic in loadbalancer.nextServer

Multiple goroutines may increment the nextServerIndex field at the same time, after its value has been clamped to the size of the list by another goroutine. This leads to an attempt to reference an item past the end of the list.

Types of Changes

bugfix

Verification

See linked issue.

This may be hard to replicate on k3s, as it requires a large number of goroutines dialing connections using a loadbalancer with multiple failed servers in the list.

Testing

Linked Issues

• [Release-1.28] - Loadbalancer may panic due to race condition when selecting a new server #10320
• RKE2 agent PANIC when controlplane quorum is lost since several hours rancher/rke2#6130

User-Facing Change

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 25.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 36.10%. Comparing base (7de7adb) to head (b1ee1bb).
Report is 2 commits behind head on release-1.28.

Files	Patch %	Lines
pkg/agent/loadbalancer/servers.go	25.00%	9 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff                @@
##           release-1.28   #10323      +/-   ##
================================================
- Coverage         42.68%   36.10%   -6.58%     
================================================
  Files               163      160       -3     
  Lines             16632    14056    -2576     
================================================
- Hits               7099     5075    -2024     
+ Misses             8274     7989     -285     
+ Partials           1259      992     -267     

Flag	Coverage Δ
e2etests	36.10% <25.00%> (-9.82%)	⬇️
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.