reloader: Expose metrics to give info about last operation result/time #4594
Conversation
philipgough
force-pushed
the
mon-1085
branch
2 times, most recently
from
415f0b9 to
a9080ee
Compare
philipgough
force-pushed
the
mon-1085
branch
3 times, most recently
from
ab4b835 to
8090288
Compare
| @@ -154,6 +158,18 @@ func New(logger log.Logger, reg prometheus.Registerer, o *Options) *Reloader { | |||
| Help: "Total number of reload requests that failed.", | |||
| }, | |||
| ), | |||
| lastReloadSuccess: promauto.With(reg).NewGauge( | |||
| prometheus.GaugeOpts{ | |||
| Name: "reloader_last_reload_successful", | |||
There was a problem hiding this comment.
Why querying counter of all reloads does not give us that? 🤔
There was a problem hiding this comment.
@bwplotka - I don't understand this sorry. We maintain a counter for total reloads and failed reloads. How would I determine from those if the latest retry was a success?
There was a problem hiding this comment.
Just looking at the mixin for bad config alert for Prometheus it achieves it using similar https://github.com/prometheus/prometheus/blob/main/documentation/prometheus-mixin/alerts.libsonnet#L8
Still unsure how we would guard against false alarm by determining without determining if the very last attempt was a success.
For example say I have a 90% failure rate on reloads, I might alert on that, although I wouldn't care if I knew the last reload attempt actually forced the reload. How would we achieve the same guard with the timestamp?
There was a problem hiding this comment.
Right - we can do that with increase(total[5m]) - increase(fails[5m]) > 0 no?
But happy to stick to standard if we are familiar with it
| ), | ||
| lastReloadSuccessTimestamp: promauto.With(reg).NewGauge( | ||
| prometheus.GaugeOpts{ | ||
| Name: "reloader_last_reload_success_timestamp_seconds", |
There was a problem hiding this comment.
This, plus counter of reloads should give us enough info, so no need for reloader_last_reload_successful maybe? (Let's avoid too many ways of doing the same thing)
Signed-off-by: Philip Gough <philip.p.gough@gmail.com>
Changes
In order to facilitate alerting on errors in the config reloader, this PR exposes four gauges which provide information on the last operation for both reloads and config applies.
These could be used similarly to
prometheus_config_last_reload_success_timestamp_secondsandprometheus_config_last_reload_successfulexposed by Prometheus.