Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[broker] Optimize TopicPolicies#maxUnackedMessagesOnConsumer with HierarchyTopicPolicies #13618

Merged
merged 3 commits into from
Jan 27, 2022

Conversation

Jason918
Copy link
Contributor

@Jason918 Jason918 commented Jan 5, 2022

Motivation

This is one of the serial topic policy optimization with HierarchyTopicPolicies.

Update topic policy with HierarchyTopicPolicies comes with these benefits:

  • All topic policy related settings will goes into AbstractTopic#topicPolicies, easier to understand, check, review, and modify.
  • Unify policy update to AbstractTopic. And easier to find which policy is not applied to non-persistent topic yet. And we can easily add support for it.
  • Unify policy value with 3 level settings (topic/namespace/broker), and priority with topic > namespace > broker. And it's easier to find which level settings is missing.
  • All values are updated at creation or by trigger. We can save some resource to update it or recalculate each time we use it.
  • etc.

Modifications

  1. Add new field maxUnackedMessagesOnConsumer in org.apache.pulsar.broker.service.AbstractTopic#topicPolicies.
  2. change maxUnackedMessages to maxUnackedMessagesEnabled in Consumer. maxUnackedMessages check is enabled only for durable cursor.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as

  • org.apache.pulsar.broker.admin.AdminApiMaxUnackedMessages

Some checklist for updating topic policy with HierarchyTopicPolicies.

  • Broker level value set. And if it's dynamic (no).
  • Namespace level value updated.
  • Topic level value updated.
  • Pre-existing unit test covers this change and updated to verify this change.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

  • no-need-doc

Code optimization.

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jan 5, 2022
@Jason918
Copy link
Contributor Author

Jason918 commented Jan 5, 2022

/pulsarbot run-failure-checks

1 similar comment
@Jason918
Copy link
Contributor Author

Jason918 commented Jan 6, 2022

/pulsarbot run-failure-checks

@Jason918 Jason918 force-pushed the tp_maxUnackedMessagesOnConsumer branch from 19484c8 to 8c2681c Compare January 6, 2022 03:56
@Jason918
Copy link
Contributor Author

@codelipenghui @315157973 PTAL

@@ -102,7 +102,7 @@

private final ConsumerStatsImpl stats;

private volatile int maxUnackedMessages;
private final boolean maxUnackedMessagesEnabled;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we update Policies, it should be possible to disable this feature. If it is set to final, how to update it later?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This maxUnackedMessagesEnabled is used for the purpose of disabling "unacked messages check" feature on consumer level for non-durable subscriptions. Following previous logic, see

int maxUnackedMessages = isDurable
? getMaxUnackedMessagesOnConsumer()
: 0;

@Jason918 Jason918 requested a review from 315157973 January 23, 2022 12:20
@Jason918 Jason918 force-pushed the tp_maxUnackedMessagesOnConsumer branch from e95c9d6 to 29ae27d Compare January 25, 2022 07:45
@Jason918
Copy link
Contributor Author

/pulsarbot run-failure-checks

@codelipenghui
Copy link
Contributor

@Jason918 Please rebase to the master branch, there is a CPP test fix in the master branch

@Jason918 Jason918 force-pushed the tp_maxUnackedMessagesOnConsumer branch from 43c4eba to 914eef7 Compare January 27, 2022 08:33
@Jason918
Copy link
Contributor Author

/pulsarbot run-failure-checks

Copy link
Contributor

@hangc0276 hangc0276 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

@Jason918 Jason918 merged commit c239d9b into apache:master Jan 27, 2022
Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-not-needed Your PR changes do not impact docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants