Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PIP 81]: Part-2 Add range set wrapper to support record dirty ledgers #15607

Merged
merged 9 commits into from
May 22, 2022

Conversation

315157973
Copy link
Contributor

Motivation

Pulsar persists the individual acknowledgments into Ledger for each subscription.
acknowledgment hole uses guava range to record.
Now a single entry is used to record these holes, but a single entry can not stores more than 5MB data.
Therefore, this PIP stores these holes in multiple entries separately. And use LRU to ensure low memory usage.
In this way, we can support almost unlimited acknowledgment hole.

Doc: https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit

We do not need to persist the data of all entries every time, only the entries whose Ledger data has changed need to be re-saved. Therefore, we add a wrapper class to record dirty pages.

Modifications

  1. Add a wrapper class
  2. Record dirty pages where add and remove

Verifying this change

RangeSetWrapperTest.java

  1. Make sure dirty pages are recorded correctly
  2. Make sure the behavior of wrapper class is the same as other RangeSets

Documentation

  • no-need-doc
    (Added configuration items, this part of the document can be automatically generated)

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label May 15, 2022
@315157973
Copy link
Contributor Author

/pulsarbot run-failure-checks

Copy link
Contributor

@codelipenghui codelipenghui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just left a comment about the config name to keep the same style with configurations we have (*Enabled)

conf/broker.conf Outdated

# If enabled, the maximum "acknowledgment holes" will not be limited and "acknowledgment holes" are stored in
# multiple entries.
enableMultiEntriesToPersistUnackedRanges=false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

persistentUnackedRangesWithMultipleEntriesEnabled=false

@codelipenghui codelipenghui added this to the 2.11.0 milestone May 17, 2022
@codelipenghui codelipenghui added type/feature The PR added a new feature or issue requested a new feature area/broker labels May 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/broker doc-not-needed Your PR changes do not impact docs type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants