Supports evenly distribute topics count when splits bundle

[codelipenghui]

Motivation

Currently, bundle split splits the bundle into two parts of the same size. When there are fewer topics, bundle split does not work well. The topic assigned to the broker according to the topic name hash value, hashing is not effective in a small number of topics bundle split.

So, this PR introduces an option(-balance-topic-count) for bundle split. When setting it to true, the given bundle splits to 2 parts, each part has the same amount of topics.

And introduce a new Load Manager implementation named org.apache.pulsar.broker.loadbalance.impl.BalanceTopicCountModularLoadManager. The new Load Manager implementation splits bundle with balance topics count, others are not different from ModularLoadManagerImpl.

Verifying this change

Test is coming

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): (no)
• The public API: (no)
• The schema: (no)
• The default values of configurations: (no)
• The wire protocol: (no)
• The rest endpoints: (no)
• The admin cli options: (no)
• Anything that affects deployment: (no)

Documentation

• Does this pull request introduce a new feature? (yes)

[jiazhai]

+1 good for small number of partitions

[sijie]

overall the direction looks good. However, I don't suggest just adding a simple boolean flag to introduce another split algorithm. This makes code maintenance pretty hard and not easy to extend in the future. Instead, I would suggest introducing an interface about BundleSplitAlgorithm. The algorithm takes NamespaceService and a Namespace Bundle as the inputs and returns a Long number as the split boundary.

interface BundleSplitAlgorithm {
    CompletableFuture<Long> split(NamespaceService service, NamespaceBundle bundle);
}

We can have two algorithms in place. One is the default implementation, splitting a bundle by dividing the key range by half; the other one is the new implementation, splitting the number by number of topics.

You can have two configurations in the broker.conf called supportedSplitAlgorithms and 'defaultBundleSplitAlgorithm'. It allows brokers to configure the supported algorithms in the broker. Then in the admin endpoint, you can add a string parameter to allow specifying a split algorithm. If users don't specify the algorithm then use the default split algorithm.

We can also add this setting to namespace policy to specify the default split algorithm for a given namespace.

Hope these make sense to you.

[codelipenghui]

@sijie Good idea, I will try to apply your comment. Thanks

[codelipenghui]

@sijie I have applied your comments, please take a look, thanks.

[sijie]

@codelipenghui the code change looks good to me. Do you mind adding documentation about this?

[codelipenghui]

@sijie I have added the description in reference-configuration. PTAL.

[tuteng]

Add label release-2.5.1, due to #6178 dependency

[Anonymitaet]

Doc has been added: https://pulsar.apache.org/docs/en/next/reference-configuration/#broker