Bug 1934163: adjust Thanos querier alerting rules #1087
Conversation
openshift-ci-robot
added
bugzilla/severity-high
|
@simonpasquier: This pull request references Bugzilla bug 1934163, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
approved
simonpasquier
force-pushed
the
bz1934163
branch
from
4876456
to
e4fc54c
Compare
|
/bugzilla refresh |
openshift-ci-robot
added
bugzilla/valid-bug
|
@simonpasquier: This pull request references Bugzilla bug 1934163, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/retest |
| @@ -97,6 +135,9 @@ local patchOrExcludeRuleGroup(group, groupSet, operation) = | |||
| [] + patchOrExcludeRuleGroup(group, groupSet[1:], operation); | |||
|
|
|||
| { | |||
| // excludedRules removes upstream rules that we don't want to carry in CMO. | |||
| rules: [ | ||
| { | ||
| alert: 'ThanosQueryHttpRequestQueryErrorRateHigh', | ||
| 'for': '1h', |
There was a problem hiding this comment.
this is based on the latest discussions in openshift/enhancements#637 (though nothing is settled). I could make it 15m to start with.
There was a problem hiding this comment.
We do have telemetry data and can estimate for duration from there. However from my quick look, 1h and 15m don't have that much of a difference.
There was a problem hiding this comment.
I've checked over the last 24 hours: 5 clusters fired the alert and it resolved for all of them after 10 to 30 minutes. In most cases no other alert was firing.
jsonnet/patch-rules.libsonnet
Outdated
| @@ -74,6 +81,37 @@ local patchedRules = [ | |||
| }, | |||
| ], | |||
| }, | |||
| { | |||
| name: 'thanos-query.rules', | |||
There was a problem hiding this comment.
For convenience purpose, would it be meaningful to update the script to be able to apply changes to all the rules of a group? In your case, applying the same duration and severity to the whole group without having to specify the name of the rules.
Currently, this can be hack around for all the alerting rules of a group by setting the alert name to "".
There was a problem hiding this comment.
yes it would make sense. Will think about it :)
|
/refresh |
|
@simonpasquier: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
This change tweaks the upstream alerts to comply with the OpenShift requirements: * The severity is set to "warning" for all alerts. * The "for" duration is increased to 1h for all alerts. * The latency alerts are removed. Signed-off-by: Simon Pasquier <spasquie@redhat.com>
simonpasquier
force-pushed
the
bz1934163
branch
from
e4fc54c
to
70e4331
Compare
| - [#1087](https://github.com/openshift/cluster-monitoring-operator/pull/1087) Decrease alert severity to "warning" for ThanosQueryHttpRequestQueryErrorRateHigh and ThanosQueryHttpRequestQueryRangeErrorRateHigh alerts. | ||
| - [#1087](https://github.com/openshift/cluster-monitoring-operator/pull/1087) Increase "for" duration to 1 hour for all Thanos query alerts. | ||
| - [#1087](https://github.com/openshift/cluster-monitoring-operator/pull/1087) Remove ThanosQueryInstantLatencyHigh and ThanosQueryRangeLatencyHigh alerts. | ||
| - [#1090](https://github.com/openshift/cluster-monitoring-operator/pull/1090) Decrease alert severity to "warning" for all Thanos sidecar alerts. |
There was a problem hiding this comment.
woops, forgot to add those, thanks!
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgrisonnet, simonpasquier The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@simonpasquier: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@simonpasquier: All pull requests linked via external trackers have merged: Bugzilla bug 1934163 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This change tweaks the upstream alerts to comply with the OpenShift
requirements:
The severity is set to "warning" for all alerts.
The "for" duration is increased to 1h for all alerts.
The latency alerts are removed.
I added CHANGELOG entry for this change.
No user facing changes, so no entry in CHANGELOG was needed.