Fix Prometheus node/cluster labels

[jage]

WHAT is this pull request doing?

• Add lavinmq_node label, hostname to differentiate nodes in cluster
• Fix lavinmq_cluster to be a common cluster name for all nodes, this will allow Prometheus grouping by cluster

Cluster name must be set on all nodes, to the same value, via the clustering_name variable.

HOW can this pull request be tested?

Start LavinMQ with --clustering-name=my-cluster-name, scrape metrics.

$ curl -s http://guest:guest@localhost:15672/metrics | grep cluster
lavinmq_identity_info{lavinmq_version="2.0.0-9-g558bb98f", lavinmq_node="8E84C28F.local", lavinmq_cluster="my-cluster-name"} 1

Example from Prometheus TSDB:

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[jage]

I have read the CLA Document and I hereby sign the CLA

[jage]

Happy to discuss other implementations, and if ok I can look at docs.

[kickster97]

looks good :)
Minor: is cluster_name better than clustering_name?

[jage]

looks good :) Minor: is cluster_name better than clustering_name?

Definitely more aesthetically pleasing but ... 🙂
... all other options are grouped under clustering, e.g. in the configuration file it's a section with [clustering] so now it's just name = ....

[kickster97]

looks good :) Minor: is cluster_name better than clustering_name?

Definitely more aesthetically pleasing but ... 🙂

... all other options are grouped under clustering, e.g. in the configuration file it's a section with [clustering] so now it's just name = ....

Ah.. makes sense, oh well!

[carlhoerberg]

How can you group on it unless you add the label to all metrics? Doing some kind of transformation somewhere? If that's case, isn't it possible to deduct the (CloudAMQP) cluster name by just dropping the last 3 chars from the node name?

[jage]

How can you group on it unless you add the label to all metrics? Doing some kind of transformation somewhere?

Exactly, you "join" the time series, example: lavinmq_queues * on(instance) group_left(lavinmq_cluster, lavinmq_node) lavinmq_identity_info to get the extra labels on the queues. Think it's pretty "standard", especially if you have high cardinality labels.

If that's case, isn't it possible to deduct the (CloudAMQP) cluster name by just dropping the last 3 chars from the node name?

Not sure that's possible in PromQL if I understand correctly, i.e. group by part of a label.

Relevant questions here:

• Is the intent to have a lavinmq_cluster label? I assumed yes since it was there., and I think LavinMQ should have have this label to avoid relabelling during scrape.
• Do you have any intent on ever having an entitiy "cluster name" for anything else then metrics? If not this could be implemented in other ways (--extra-metrics-labels or something like that)

[carlhoerberg]

Relevant questions here:

• Is the intent to have a lavinmq_cluster label? I assumed yes since it was there., and I think LavinMQ should have have this label to avoid relabelling during scrape.

• Do you have any intent on ever having an entitiy "cluster name" for anything else then metrics? If not this could be implemented in other ways (--extra-metrics-labels or something like that)

Yeah, nothing comes to mind now how we could use this for anything else, hence my hesitation. The label makes sense, but --extra-metrics-label lavinmq_cluster=red-panda --extra-metrics-label foo=bar sounded like a more generic idea!