Wait adaptive duration per unready replicas between retries

[wongni]

Issue #, if available:

Description of changes:
Currently it waits zero seconds between retries when waiting for all machine deployment replicas are ready. This PR introduces a wait time adaptive to unready replicas between retries.

Testing (if applicable):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[wongni]

/cc @chrisdoherty4 @g-gaston

[codecov]

Codecov Report

Merging #2381 (1163016) into main (1484135) will decrease coverage by 1.93%.
The diff coverage is 96.77%.

❗ Current head 1163016 differs from pull request most recent head 013035c. Consider uploading reports for the commit 013035c to get more accurate results

@@            Coverage Diff             @@
##             main    #2381      +/-   ##
==========================================
- Coverage   56.09%   54.16%   -1.94%     
==========================================
  Files         303      304       +1     
  Lines       24657    26164    +1507     
==========================================
+ Hits        13831    14171     +340     
- Misses       9533    10677    +1144     
- Partials     1293     1316      +23     

Impacted Files	Coverage Δ
pkg/clustermanager/cluster_manager.go	54.49% <93.33%> (+1.32%)	⬆️
pkg/executables/kubectl.go	40.75% <100.00%> (+2.33%)	⬆️
pkg/api/v1alpha1/zz_generated.deepcopy.go	19.98% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1484135...013035c. Read the comment docs.

[g-gaston]

Looks good to me!
If we can increase the test coverage for the diff a bit, it's ready to go

[g-gaston]

/lgtm
we just need to get the test coverage a bit higher

[g-gaston]

/lgtm
/approve

[eks-distro-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: g-gaston

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [g-gaston]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment