[CLE] Parallelize lease candidate ping order

[Jefftree]

In the leader election controller: https://github.com/kubernetes/kubernetes/blob/master/pkg/controlplane/controller/leaderelection/leaderelection_controller.go#L267-L293, candidates are iterated through sequentially and LeaseCandidates.Update() is a blocking operation that is also called sequentially. This can lead to cases where certain candidates do not have enough time to respond. We should parallelize this entire operation and iterate through the list in a random order to ensure every candidate has a fair chance of responding.

/cc @Henrywu573
/triage accepted
/sig api-machinery

[tchap]

Sorry, linked this issue by accident when playing with some simple issues to fix in my spare time.

Do you really need to randomize the order? The concurrent requests don't block each other. I guess something like tchap@415c889 could suffice?

[Henrywu573]

/assign

[Jefftree]

Sorry, linked this issue by accident when playing with some simple issues to fix in my spare time.

Do you really need to randomize the order? The concurrent requests don't block each other. I guess something like tchap@415c889 could suffice?

cc @sttts. I think it makes sense that randomization is not strictly necessary if we parallelize. We should be able to guarantee that each candidate has a fair chance of acking with just the parallelization.