Eliminated race condition in E2E tests

[RyanDevlin]

Fix

This PR fixes a bug in the E2E tests involving a race condition that can occur in some Kind clusters.

Description

This bug occurs because the first E2E test (currently TestRemoveDuplicates) lists the nodes. Since the E2E automation builds a Kind cluster and then immediately executes the tests, the nodes may or may not be ready at the time the first test lists them. This can cause unpredictable behavior, because if one or more nodes are not ready when the test lists them, the nodes the test case sees will be tainted with node.kubernetes.io/not-ready. This ultimately results in the Deschedulder avoiding evictions because it only has knowledge of tainted nodes, which causes the test case to fail.

Solution

To fix this, I've added a waitForNodesReady() function which is called by the initializeClient() setup routine. This allows the test cases to wait until all nodes are Ready before running. The waiting period is typically only a second or two.

[k8s-ci-robot]

Hi @RyanDevlin. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[RyanDevlin]

@ingvagabund Moved from #636 to here.

[ingvagabund]

/approve
/lgtm
/ok-to-test

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ingvagabund, RyanDevlin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ingvagabund]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[RyanDevlin]

/retest

[RyanDevlin]

@ingvagabund It looks like the descheduler-helm-test is failing to build the control plane? Any idea why this might occur?

[damemi]

/retest
I saw a similar helm test failure in another PR... don't know what's up with that but trying another retest to see if it's a flake. Don't think we merged anything recently that would break it