Implement Retryable tasks

[xmudrii]

What this PR does / why we need it:

This PR implements retryable tasks. Some tasks are ran only once while some are ran 3 times. This package is based on the k8s.io/apimachinery/util/wait package and the ExponentialBackoff function.

This is required for proper upgrades support from 1.13 to 1.14 and to stabilize the CI.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
xref #269

Release note:

NONE

/assign @kron4eg

[xmudrii]

/test pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

[kron4eg]

/lgtm
/approve

[kubermatic-bot]

LGTM label has been added.

Git tree hash: 17b5d7daa4d3d7dc131e6444bdf46269cb22d968

[kubermatic-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kron4eg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kron4eg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[xmudrii]

/test pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0
This is related to E2E tests and not to the upgrade logic itself. I'll create a PR to fix this.

[xmudrii]

/retest

[xmudrii]

/retest

[xmudrii]

The cluster never got provisioned.
/test pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

[xmudrii]

/retest

[xmudrii]

/retest

[xmudrii]

/override pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

[kubermatic-bot]

@xmudrii: Overrode contexts on behalf of xmudrii: pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

In response to this:

/override pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[xmudrii]

/retest

[kubermatic-bot]

@xmudrii: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0	b848f23	link	/test pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

Full PR test history

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[xmudrii]

This is a known flake and we had it for a long time now. For some reason API seems to be slower than usual and it takes more time to replace the worker nodes than expected. If you check nodes in the cluster, you'll see that they're getting replaced, but tests timeout by the time the third node should be replaced.

Because this is not a release blocking flake and that all other tests have many green runs, I'm going to skip this test.
/override pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

[kubermatic-bot]

@xmudrii: Overrode contexts on behalf of xmudrii: pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

In response to this:

This is a known flake and we had it for a long time now. For some reason API seems to be slower than usual and it takes more time to replace the worker nodes nodes than expected. If you check nodes in the cluster, you'll see that they're getting replaced, but tests timeout by the time the third node should be replaced.

Because this is not a release blocking flake and that tests all other tests have many green runs, I'm going to skip this test.
/override pull-kubeone-e2e-digitalocean-upgrade-1.13.5-1.14.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.