wip: expose consumer group lag as prometheus metrics

[JamieAP]

No description provided.

[baakind]

Is there a reason why this PR died? We are currently using Burrow to monitor lag and partition-status, and we do want to monitor this using Prometheus. I can see that it is a bit outdated, but I'm curious if there is a reason why it just died. @toddpalino @JamieAP

[toddpalino]

Not sure why this dropped out, but it probably needs a do-over for 1.0 anyways. The one caveat I would say on that is that if the write to prometheus is straight HTTP, it should probably be implemented using the HTTP notifier and an example template and config provided (which can be added to the documentation).

We do something like this internally with metrics, but I haven't published a sample template yet.

[varun06]

Yeah, I am also looking for prom metrics in burrow.

[daodennis-zz]

There is some existing work too that maybe we can borrow for Burrow in the burrow_exporter

Other projects like Kubernetes, etcd, and Docker utilize Prometheus instrumentation. Also, organizations like Cloudflare who use both Kafka and Prometheus like ours are not an isolated intersection. There is a formidable and active operator community behind Prometheus as well.
Anyway...

jirwin's exporter has these metrics:

KafkaConsumerPartitionCurrentOffset
KafkaConsumerPartitionMaxOffset
KafkaConsumerTotalLag

Also, it would be nice to expose operational metrics, most places where there's a logging statement could use a metric. @JamieAP are you planning on shoring this PR up for 1.0 by chance?

[varun06]

Don't want to ruin holiday week, but if any help needed for this PR, I can find some time. Will really appreciate if this get merged sooner than later.

[JamieAP]

Sorry for the delay. I abandoned this (and forgot to tidy the PR) in favour of building something slightly more specific to my use case. I'm unlikely to have any time to work on this PR in the next couple of weeks so please feel free to pick it up.

@toddpalino Prometheus uses a pull model. The prometheus backend makes a request to an endpoint to fetch metrics rather than a service making a request to a prometheus endpoint to write them. The HTTP notifier would be no help here.