bookies hosts by metrics grow exponentially.

[thinker0]

BUG REPORT
There are many, many, many Prometheus Metrics names.

Describe the bug

^(bookkeeper_client_per_channel_bookie_client|bookkeeper_server_BookieHighPriorityThread|replication_bookkeeper_client_per_channel_bookie_client)_hostnames_port_.*

bookies hosts by metrics grow exponentially.

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
Prometheus OOM (200Gbyte)

Screenshots

Additional context

Add any other context about the problem here.

[eolivelli]

which version of BK are you using ?
how many bookies ?
do bookies have a dynamic address ?

[thinker0]

• pulsar-2.7.2
• 120ea bookies
• yes (apache mesos/aurora)
  

Change the metrics as shown below.

bookkeeper_client_per_channel_bookie_client...{instance="hostname:port"}

[thinker0]

Like this ? apache/incubator-heron#3493

[sursingh]

I think this has already been addressed: https://github.com/apache/bookkeeper/blame/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java#L407

4.14.1 release should have this change

[thinker0]

@sursingh Is it compatible with Pulsar 2.7.2 version?

[sursingh]

2.7.2 is using bookkeeper 4.12.x so doesn't have this change. You may need to backport this to 4.12.x bookkeeper.

[thinker0]

https://github.com/apache/bookkeeper/releases/tag/release-4.14.1

Nothing binary ?

[sursingh]

Binary jars can be downloaded from maven. https://mvnrepository.com/artifact/org.apache.bookkeeper/bookkeeper-server

[sijie]

@thinker0 Can you create an issue in Pulsar? We can upgrade the bookkeeper version for Pulsar 2.7.3 release.

[hezhangjian]

has been solved in 4.14.1