-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PIP 81]: Part-2 Add range set wrapper to support record dirty ledgers #15607
Conversation
/pulsarbot run-failure-checks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just left a comment about the config name to keep the same style with configurations we have (*Enabled)
conf/broker.conf
Outdated
|
||
# If enabled, the maximum "acknowledgment holes" will not be limited and "acknowledgment holes" are stored in | ||
# multiple entries. | ||
enableMultiEntriesToPersistUnackedRanges=false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
persistentUnackedRangesWithMultipleEntriesEnabled=false
Motivation
Pulsar persists the individual acknowledgments into Ledger for each subscription.
acknowledgment hole uses guava range to record.
Now a single entry is used to record these holes, but a single entry can not stores more than 5MB data.
Therefore, this PIP stores these holes in multiple entries separately. And use LRU to ensure low memory usage.
In this way, we can support almost unlimited acknowledgment hole.
Doc: https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit
We do not need to persist the data of all entries every time, only the entries whose Ledger data has changed need to be re-saved. Therefore, we add a wrapper class to record dirty pages.
Modifications
add
andremove
Verifying this change
RangeSetWrapperTest.java
Documentation
no-need-doc
(Added configuration items, this part of the document can be automatically generated)