Pre-populated metrics for statsd and similar push models

[vorner]

Hello

I'm not sure this would belong into dipstick itself, so I'm asking what you think about it.

With the statsd collector, and likely with other push ones too, if metric is created but not used, it isn't sent at all. This can happen to a rare error metric. This has the downside of any plotting frontend not knowing about that metric (like grafana), creating alerts and graphs is therefore „blind“, which is prone to typos, etc ‒ and if the error eventually comes, it can be missed because of the typo.

The hack we do is adding an .init method to all the metric types and using it once we create the metric, to send one instance of some inert value (eg. 0 for counter).

Would it make sense to upstream it here? Or even do it automatically, without calling the method? Or doing it only for the push model sinks?

[fralalonde]

I've also implemented such "keepalive" functionality before and it definitely falls within the scope of dipstick to prevent discontinuity in the stream of metric values. I believe is actually an essential feature! This is also related to #8 as implementation it would solve both cases.

I'd try implementing this as a decorator, either as an extension of the existing caching layer or in a similar but separate wrapper. This would give the wrapped output a memory of the metrics that have been created, and allow to remember the last seen value for each (if any). Filling the blanks by sending safe values should be configurable to be triggered either by direct scheduling or by an external flush event (which might itself be running on schedule).

This is also related to the on_flush() functionality I was proposing in the pending #50 which I intend to merge shortly.

What value would be sent for a timer that has never been used?

[vorner]

What value would be sent for a timer that has never been used?

No idea about the correct one to send. We send 0, but that one is wrong :-(. It isn't causing any pain for us, because for us the timers are not rare and the real ones simply outweight the fake, but I don't know what I'd put in there for a library.

[fralalonde]

See #61 for tentative implementation of this.
Specifically https://github.com/fralalonde/dipstick/blob/feature/retain-last/src/cache/gapless_in.rs

[vorner]

Thanks, I'll have a look soon.

[vorner]

It's probably slightly different use case than what I've had in my mind, but it would probably work too, so yes, this would be nice to have.

I just wonder if this can skew the outputs in somehow misleading ways ‒ for some graphs, I actually want to see the gaps if there are any, I think. But I guess that's mostly a documentation issue.

I guess it's not easily possible to make some metrics gapless and some others with gaps and have them side by side? I could probably create two different trees of metrics, one with this adaptor, one without, but that could become unergonomic fast.