Consider all peers as potential candidates during pull-request in case of offline nodes

[AshwinSekar]

Problem

When we receive contact info from an offline node in our list of peers, we assign it a much higher weight than other peers because we have not sent a pull request to it yet. This causes the sampling to always select only this peer to send pull requests to. However the ping check never allows us to send a pull request and update the timestamp, as the node is offline. Thus we never get a chance to send pull requests to other nodes.

Summary of Changes

After assigning the weights, instead of performing random sampling we use a weighted shuffle to ensure that if the higher weighted nodes fail the ping check we can still send pull requests to the other nodes.
Also reenables test_no_optimistic_confirmation_violation_with_tower and test_optimistic_confirmation_violation_without_tower

Fixes #18279

[sakridge]

All peers will be all 500 or 1000 nodes?

[AshwinSekar]

Sorry about the wording we are still only sending 1 pull request.

Instead of taking 2 * # nodes samples of the peers and finding the first available to send a pull request to, we do a weighted shuffle amongst all the peers to find the first available. The idea is that if offline nodes have a large weight, all the samples could result in offline nodes which prevents us from sending any pull requests.

[behzadnouri]

LGTM, thanks for debugging the local-cluster tests

Pull request has been modified.

[codecov]

Codecov Report

Merging #18333 (3f43f09) into master (8d9a6de) will decrease coverage by 0.0%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##           master   #18333     +/-   ##
=========================================
- Coverage    82.4%    82.3%   -0.1%     
=========================================
  Files         434      434             
  Lines      121329   121335      +6     
=========================================
- Hits        99978    99953     -25     
- Misses      21351    21382     +31     

[CriesofCarrots]

v1.7 backport?