Fix issue caused by sole server marked as failed under load

[brandond]

Proposed Changes

Fix issue caused by sole server marked as failed under load.

If health checks are failing for all servers, make a second pass through the server list with health-checks ignored before returning failure.

Types of Changes

bugfix

Verification

See linked issue - probably hard/impossible to reproduce on demand due to failure conditions

Testing

Linked Issues

• Health checks may remove all available servers from load-balancer under high load #10240

User-Facing Change

The embedded load-balancer will now fall back to trying all servers with health-checks ignored, if all servers have been marked unavailable due to failed health checks.

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 41.73%. Comparing base (ed23a2b) to head (ca39614).

Files	Patch %	Lines
pkg/agent/loadbalancer/loadbalancer.go	83.33%	0 Missing and 1 partial ⚠️
pkg/etcd/etcdproxy.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10241      +/-   ##
==========================================
- Coverage   47.83%   41.73%   -6.11%     
==========================================
  Files         177      177              
  Lines       14765    14772       +7     
==========================================
- Hits         7063     6165     -898     
- Misses       6358     7428    +1070     
+ Partials     1344     1179     -165     

Flag	Coverage Δ
e2etests	36.45% <83.33%> (-9.46%)	⬇️
inttests	37.08% <41.66%> (+<0.01%)	⬆️
unittests	11.34% <75.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.