Update mixin for TempoIngesterFlushes thresholds #1354
Conversation
zalegrala
requested review from
joe-elliott,
annanay25,
mdisibio,
dgzlopes,
mapno and
yvrhdn
as code owners
zalegrala
force-pushed
the
unhealthyIngesterFlush
branch
2 times, most recently
from
668a33d to
cc59827
Compare
zalegrala
changed the title
| severity: 'warning', | ||
| }, | ||
| annotations: { | ||
| message: 'Greater than %s flushes have failed in the past hour.' % $._config.alerts.flushes_per_hour_failed, |
There was a problem hiding this comment.
I think it would be good to tweak the message so the warning and critical are different. Don't have a recommendation though... any ideas?
There was a problem hiding this comment.
Good idea, I'll wordsmith around a little.
There was a problem hiding this comment.
Updated a little. How's that read to you now?
zalegrala
force-pushed
the
unhealthyIngesterFlush
branch
from
cc59827 to
2df00cf
Compare
| expr: ||| | ||
| sum by (%s) (increase(tempo_ingester_failed_flushes_total{}[1h])) > %s and | ||
| sum by (%s) (increase(tempo_ingester_failed_flushes_total{}[5m])) > 0 | ||
| sum by (%s) (increase(tempo_ingester_flush_failed_retries_total{}[1h])) > %s and |
There was a problem hiding this comment.
what do we think about keying the "unhealthy" warning alert on the old metric and the "failing" critical alert on the retries metric?
then for consistency keep the for: 5m on both?
There was a problem hiding this comment.
That could be reasonable. The goal being to always know when a flush failed at a warning level, but only get paged when a retry fails for the 5m duration?
There was a problem hiding this comment.
Yeah, that's what i was thinking. 👍
There was a problem hiding this comment.
Okay, I've made the update.
zalegrala
force-pushed
the
unhealthyIngesterFlush
branch
from
70b2613 to
0ca1ccd
Compare
What this PR does:
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]