Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Topic-Level Metrics That Are Partition-Inclusive #11432

Closed
evfurman opened this issue Jul 22, 2021 · 1 comment · Fixed by #12225
Closed

Topic-Level Metrics That Are Partition-Inclusive #11432

evfurman opened this issue Jul 22, 2021 · 1 comment · Fixed by #12225
Assignees
@codelipenghui
Labels
release/2.8.2 release/2.9.0 type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Milestone
2.9.0

Comments

@evfurman
Copy link

Is your enhancement request related to a problem? Please describe.
We'd love to be able to get topic-level metrics that are partition-inclusive. We use the openmetrics integration in Datadog but when we break out the metric by topic it includes the partition number in the topic name.

Describe the solution you'd like
Would like topic tag to be the actual topic (without partition number) and then have a separate partition tag

CURRENT
this-pulsar-topic-partition-1

DESIRED
this-pulsar-topic

Describe alternatives you've considered
We could write a script to rewrite the metrics and publish them to prometheus but a native solution within pulsar itself would be much cleaner.
@evfurman evfurman added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Jul 22, 2021
@sijie sijie mentioned this issue Jul 22, 2021
Closed
@Technoboy-
Copy link
Contributor

Currenly, pulsar broker owns the topic based on namespace bundles, that means : a topic with 3 partitions may own by different brokers, so topic level metrics is hard to aggregate between brokers.

codelipenghui added a commit to codelipenghui/incubator-pulsar that referenced this issue Sep 28, 2021
@codelipenghui
[Metrics] Add support for splitting topic and partition label in Prom…
84143ca 
…etheus

Fix: apache#11432

Currently, we are only expose the partition name for the topic label in Prometheus metrics, which is difficult to
have an aggregated metrics for a partitioned topic.

Before this change, we can only get (topic=xxx-partition-0) in the metrics. After this change, we can get 2 labels (topic=xxx, partition=0).
By default, the broker expose the single tag for topic. It need to change `splitTopicAndPartitionLabelInPrometheus=true` in the broker.conf

New tests added.
@codelipenghui codelipenghui mentioned this issue Sep 28, 2021
Merged
@codelipenghui codelipenghui self-assigned this Sep 30, 2021
@codelipenghui codelipenghui added this to the 2.9.0 milestone Sep 30, 2021
merlimat pushed a commit that referenced this issue Sep 30, 2021
@codelipenghui
[Metrics] Add support for splitting topic and partition label in Prom…
039079e 
…etheus (#12225)

* [Metrics] Add support for splitting topic and partition label in Prometheus

Fix: #11432

Currently, we are only expose the partition name for the topic label in Prometheus metrics, which is difficult to
have an aggregated metrics for a partitioned topic.

Before this change, we can only get (topic=xxx-partition-0) in the metrics. After this change, we can get 2 labels (topic=xxx, partition=0).
By default, the broker expose the single tag for topic. It need to change `splitTopicAndPartitionLabelInPrometheus=true` in the broker.conf

New tests added.

* Fix checkstyle.
codelipenghui added a commit that referenced this issue Oct 6, 2021
@codelipenghui
[Metrics] Add support for splitting topic and partition label in Prom…
5fad80c 
…etheus (#12225)

* [Metrics] Add support for splitting topic and partition label in Prometheus

Fix: #11432

Currently, we are only expose the partition name for the topic label in Prometheus metrics, which is difficult to
have an aggregated metrics for a partitioned topic.

Before this change, we can only get (topic=xxx-partition-0) in the metrics. After this change, we can get 2 labels (topic=xxx, partition=0).
By default, the broker expose the single tag for topic. It need to change `splitTopicAndPartitionLabelInPrometheus=true` in the broker.conf

New tests added.

* Fix checkstyle.

(cherry picked from commit 039079e)
nicoloboschi pushed a commit to datastax/pulsar that referenced this issue Jan 25, 2022
@codelipenghui @nicoloboschi
[Metrics] Add support for splitting topic and partition label in Prom…
2dd8093 
…etheus (apache#12225)

* [Metrics] Add support for splitting topic and partition label in Prometheus

Fix: apache#11432

Currently, we are only expose the partition name for the topic label in Prometheus metrics, which is difficult to
have an aggregated metrics for a partitioned topic.

Before this change, we can only get (topic=xxx-partition-0) in the metrics. After this change, we can get 2 labels (topic=xxx, partition=0).
By default, the broker expose the single tag for topic. It need to change `splitTopicAndPartitionLabelInPrometheus=true` in the broker.conf

New tests added.

* Fix checkstyle.

(cherry picked from commit 039079e)
(cherry picked from commit 5fad80c)
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this issue Mar 18, 2022
@codelipenghui @ciaocloud
[Metrics] Add support for splitting topic and partition label in Prom…
c7e6b9e 
…etheus (apache#12225)

* [Metrics] Add support for splitting topic and partition label in Prometheus

Fix: apache#11432

Currently, we are only expose the partition name for the topic label in Prometheus metrics, which is difficult to
have an aggregated metrics for a partitioned topic.

Before this change, we can only get (topic=xxx-partition-0) in the metrics. After this change, we can get 2 labels (topic=xxx, partition=0).
By default, the broker expose the single tag for topic. It need to change `splitTopicAndPartitionLabelInPrometheus=true` in the broker.conf

New tests added.

* Fix checkstyle.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@codelipenghui codelipenghui

Labels
release/2.8.2 release/2.9.0 type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Projects
None yet
Milestone
2.9.0
4 participants
@Technoboy- @evfurman @eolivelli @codelipenghui