BUG 1858400: [Performance] Lease refresh period for machine-api-controllers is too high, causes heavy writes to etcd at idle #649
Conversation
The machine-api-controller components are refreshing their lease more than all other components combined. Bringing this to 90s each, will decrease etcd writes at idle.
The machine-api-controller components are refreshing their lease more than all other components combined. Bringing this to 90s each, will decrease etcd writes at idle.
The machine-api-controller components are refreshing their lease more than all other components combined. Bringing this to 90s each, will decrease etcd writes at idle.
The machine-api-controller components are refreshing their lease more than all other components combined. Bringing this to 90s each, will decrease etcd writes at idle.
openshift-ci-robot
added
bugzilla/severity-high
|
@Danil-Grigorev: This pull request references Bugzilla bug 1858400, which is valid. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@Danil-Grigorev: This pull request references Bugzilla bug 1858400, which is valid. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: enxebre The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
|
@Danil-Grigorev can we also please enforce this via https://github.com/openshift/machine-api-operator/blob/master/pkg/operator/sync.go#L425 |
Agree, this will help when MAO will be revendored in providers next time - 828c453 |
|
/lgtm |
pkg/operator/sync.go
Outdated
| @@ -423,6 +423,7 @@ func newContainers(config *OperatorConfig, features map[string]bool) []corev1.Co | |||
| "--logtostderr=true", | |||
| "--v=3", | |||
| "--leader-elect=true", | |||
| "-leader-elect-lease-duration=90s", | |||
There was a problem hiding this comment.
why only one dash here while any other command has --?
There was a problem hiding this comment.
Copy pasting from CLI is an unhealthy habit 😅Even when it works both ways, will fix in a second
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/hold |
openshift-ci-robot
added
the
do-not-merge/hold
Enforce this value in MAO deployment will ensure the value won't be overritten by changed CLI defaults in the future.
Danil-Grigorev
force-pushed
the
leader-election
branch
from
828c453
to
761bcd5
Compare
|
/lgtm |
|
/hold cancel |
openshift-ci-robot
removed
the
do-not-merge/hold
|
/retest |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@Danil-Grigorev: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@Danil-Grigorev: Some pull requests linked via external trackers have merged: openshift/machine-api-operator#649, openshift/cluster-api-provider-ovirt#56. The following pull requests linked via external trackers have not merged:
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
The machine-api-controller components are refreshing their lease more
than all other components combined. Bringing this to 90s each, will
decrease etcd writes at idle.
All relevant PRs: