[feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace #16901
Conversation
github-actions
bot
added
the
doc-required
|
/pulsarbot rerun-failure-checks |
| CompletableFuture<List<String>> childrenOfNamespace = store.getChildren(path); | ||
| CompletableFuture<List<String>> topics = childrenOfNamespace.thenCompose(children -> | ||
| !children.isEmpty() && children.get(0).startsWith(TopicName.BUCKET_ID_PERFIX) | ||
| ? children.stream().map(bucket -> store.getChildren(path + "/" + bucket)) | ||
| .collect(FutureUtil.toList()) | ||
| : CompletableFuture.completedFuture(children)); |
There was a problem hiding this comment.
This looks identical to L76-81, could they share a common helper method?
There was a problem hiding this comment.
Nice catch. I did not realize this. I extracted the common part into a method.
| return pulsar.getPulsarResources().getTopicResources().listPersistentTopicsAsync(namespaceName); | ||
| return pulsar.getPulsarResources().getTopicResources().listPersistentTopicsAsync(namespaceName) | ||
| .thenApply(x -> { | ||
| LOG.info("getListOfPersistentTopics: " + x); |
There was a problem hiding this comment.
Does this need to be info?
(also x -> topics)
There was a problem hiding this comment.
I don't think it needs to be logged at all. Reverted this part.
|
/pulsarbot rerun-failure-checks |
|
/pulsarbot rerun-failure-checks |
|
/pulsarbot rerun-failure-checks |
|
/pulsarbot rerun-failure-checks |
|
/pulsarbot rerun pulsar-ci-checks-completed |
|
/pulsarbot run-failure-checks |
1 similar comment
|
/pulsarbot run-failure-checks |
|
@merlimat , please take a look |
|
The pr had no activity for 30 days, mark with Stale label. |
|
@merlimat, I don't think I will have time to work on this anytime soon. Feel free to take over if this is still relevant. |
|
Thanks for your contribution @andrasbeni! It seems the community doesn't give this PR a review in time. I add this PR in my backlog to see if it's still relevant. Perhaps ping you for reviewing if so. |
Fixes #15254
Motivation
The number of topics in a namespace is limited by ZooKeeper when it is used as the local store. This is because listing topics requires all topic names to fit in ZK's buffer. By introducing an intermediate layer between the namespace node and topics, a namespace can have more topics.
Modifications
number_of_bucketsdefaulting to 1 (i.e. no buckets)Verifying this change
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below or label this PR directly.
Need to update docs?
doc-required(Your PR needs to update docs and you will update later)
Matching PR in forked repository
PR in forked repository: andrasbeni#4