Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add api to increase partitions of existing non-global partitioned-topic #299

Merged
merged 2 commits into from
May 23, 2017
Merged

add api to increase partitions of existing non-global partitioned-topic #299

merged 2 commits into from
May 23, 2017

Conversation

rdhabalia
Copy link
Contributor

Motivation

Addressing #291.

Modifications

  • It provides REST/Admin API to increment number of partitions of existing non-global topic.
  • It creates new subscriptions under new partitioned-topics and updates partitioned-topics metadata in zk.
  • It doesn't restart connected producers and consumers as partitioned-metadata lookup only happens when client creates partitioned producer and consumer and reconnection doesn't help to see new partitioned-metadata. So, client-app can restart producer/consumer to see the partitioned-metadata change.

Result

pulsar can support incrementing of partitions

@rdhabalia rdhabalia added the type/feature The PR added a new feature or issue requested a new feature label Mar 20, 2017
@rdhabalia rdhabalia added this to the 1.17 milestone Mar 20, 2017
@rdhabalia rdhabalia self-assigned this Mar 20, 2017
@merlimat merlimat modified the milestones: 1.17, 1.18 Mar 31, 2017
@merlimat
Copy link
Contributor

merlimat commented Apr 6, 2017

The change looks good for a quick approach to the issue. The only concern that I have is that there is no way to retain ordering when doing this operation, since the switch from the old to new partitions is not coordinated.

@msb-at-yahoo
Copy link
Contributor

The violation of the ordering behavior should be included in the documentation.

@rdhabalia
Copy link
Contributor Author

The only concern that I have is that there is no way to retain ordering when doing this operation, since the switch from the old to new partitions is not coordinated.

That's correct. Here after increasing partitions, disconnecting producers/consumer will not help because they again connect to individual partition-topic and client has to restart application to create producer/consumer for newly created partitioned because that's the only time when PulsarClient fetches new partition-metadata and creates producers for them. And client restart has to be coordinated.
So, in that case as @msb-at-yahoo mentioned: we can document it at api and admin-tool.
any thought?

@rdhabalia
Copy link
Contributor Author

@merlimat updated the doc for ordering-violation behavior.

merlimat
merlimat approved these changes May 12, 2017
View reviewed changes
Copy link
Contributor

@merlimat merlimat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good. Just a couple of minor conflicts with master

@rdhabalia rdhabalia force-pushed the partition-consumer branch from b121bf0 to 9603e7a Compare May 19, 2017 00:52
@rdhabalia
Copy link
Contributor Author

yes, rebased with master.

@merlimat merlimat merged commit 5ba05cc into apache:master May 23, 2017
@rdhabalia rdhabalia deleted the partition-consumer branch June 21, 2017 18:51
hrsakai pushed a commit to hrsakai/pulsar that referenced this pull request Dec 10, 2020
@wolfstudy
Support producer name for Message (apache#299)
f7ca912 
* Support producer name for Message

Signed-off-by: xiaolong.ran <rxl@apache.org>

* add test case for producer name

Signed-off-by: xiaolong.ran <rxl@apache.org>

* fix ci error

Signed-off-by: xiaolong.ran <rxl@apache.org>

* fix comments

Signed-off-by: xiaolong.ran <rxl@apache.org>

* fix comments

Signed-off-by: xiaolong.ran <rxl@apache.org>
hangc0276 pushed a commit to hangc0276/pulsar that referenced this pull request May 26, 2021
@BewareMyPower
Fix PulsarEntryFormatter not handle null value correctly (apache#305)
9b70a69 
Fixes apache#299

PulsarEntryFormatter doesn't handle null value correctly now. If a message with null value was sent by a producer, the consumer will receive a message with an empty string.

This PR fixes the bug and adds to tests to verify KoP could handle the null value no matter what the entry format is.
@xiaotongwang1 xiaotongwang1 mentioned this pull request Aug 4, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@merlimat merlimat merlimat approved these changes

Assignees

@rdhabalia rdhabalia

Labels
type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Milestone
1.18
Development

Successfully merging this pull request may close these issues.
3 participants
@rdhabalia @merlimat @msb-at-yahoo