[pulsar-broker] Support configuration to rate-limit dispatching on batch message #12294
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdhabalia
added
area/broker
doc-required
rdhabalia
requested review from
merlimat,
sijie,
jiazhai,
eolivelli,
codelipenghui and
congbobo184
|
@rdhabalia feel free to ping me if you need a doc review. |
eolivelli
approved these changes
Oct 13, 2021
|
this change doesn't change default behavior because it comes with the config. also it's addressing serious issue for batch messages so, I don't see any issue for merging for 2.9 |
wangjialing218
pushed a commit
to wangjialing218/pulsar
that referenced
this pull request
Oct 21, 2021
Anonymitaet
added
doc-complete
eolivelli
pushed a commit
to eolivelli/pulsar
that referenced
this pull request
Nov 29, 2021
2 tasks
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
We introduced dispatch throttling as part of PIP-3 which helps brokers to control fan-out and prevent broker's instability due to large message dispatch. However, current throttling applies rate-limiting based on a number of messages within batch-message. Because of that overall, broker dispatches less number of messages even though the broker has enough bandwidth if the given batch message has a large number of messages eg: 100 or 1000 msgs per batch. Also, various topics under a namespace have messages with various batch sizes and it makes unpredictable throttling for all topics under a namespace and it causes backlog building for some of the topics.
Therefore, the broker should have a configuration to count a number of batch messages rather than total messages within each batch message during rate limiting.
Modification
Add configuration
dispatchThrottlingOnBatchMessageEnabledto support rate-limiting dispatching on the batch messages rather than individual messages within batch messages. By default it will be disabled so, it won't change the existing behavior of the broker.Result
Avoid backlog building for topics in a namespace with different sizes of batch messages during dispatch rate-limiting.