BUG 1858400: [Performance] Lease refresh period for machine-api-controllers is too high, causes heavy writes to etcd at idle

[elmiko]

Prevent machine controllers from writing in etcd at idle too often by
setting 120s lease, 20s retry and 110s deadline on all renewals. Higher
values cause tests to flake.

Inspired by openshift/cloud-credential-operator#231
Depends on openshift/machine-api-operator#675

[openshift-ci-robot]

@elmiko: This pull request references Bugzilla bug 1858400, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

BUG 1858400: [Performance] Lease refresh period for machine-api-controllers is too high, causes heavy writes to etcd at idle

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@elmiko: This pull request references Bugzilla bug 1858400, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

BUG 1858400: [Performance] Lease refresh period for machine-api-controllers is too high, causes heavy writes to etcd at idle

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[elmiko]

/bugzilla refresh

[openshift-ci-robot]

@elmiko: This pull request references Bugzilla bug 1858400, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.6.0) matches configured target release for branch (4.6.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Danil-Grigorev]

@dhellmann Do you have a moment to review? We'd need to backport this implementation into 4.5, but before that, this BZ should be fixed.

[elmiko]

@dhellmann Do you have a moment to review? We'd need to backport this implementation into 4.5, but before that, this BZ should be fixed.

i was going to add the backports as my next step, i had been using this bug to report under https://bugzilla.redhat.com/show_bug.cgi?id=1861896

basically i have been backporting the leader elect mechanisms and bringing all these timing changes at once.

[stbenjam]

/retest

[dhellmann]

The change looks fine. It sounds like there's a known issue with the CI job that isn't related to this PR.

/approve

[elmiko]

/retest

[Danil-Grigorev]

/lgtm

[elmiko]

updating with 90s retry time
/hold cancel

[openshift-ci-robot]

@elmiko: This pull request references Bugzilla bug 1858400, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.6.0) matches configured target release for branch (4.6.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

BUG 1858400: [Performance] Lease refresh period for machine-api-controllers is too high, causes heavy writes to etcd at idle

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[elmiko]

after some discussion with the team we have agreed that 20s retry period is acceptable for now. i have fixed the pr
/hold cancel

[openshift-ci-robot]

@elmiko: This pull request references Bugzilla bug 1858400, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.6.0) matches configured target release for branch (4.6.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

BUG 1858400: [Performance] Lease refresh period for machine-api-controllers is too high, causes heavy writes to etcd at idle

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Danil-Grigorev]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Danil-Grigorev, dhellmann, elmiko

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dhellmann]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@elmiko: Some pull requests linked via external trackers have merged:

• openshift/machine-api-operator#675
• openshift/machine-api-operator#649

The following pull requests linked via external trackers have not merged:

• openshift/cluster-api-provider-ovirt#66 is open

These pull request must merge or be unlinked from the Bugzilla bug in order for it to move to the next state.

Bugzilla bug 1858400 has not been moved to the MODIFIED state.

In response to this:

BUG 1858400: [Performance] Lease refresh period for machine-api-controllers is too high, causes heavy writes to etcd at idle

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.