Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KAFKA-3949: Avoid race condition when subscription changes during rebalance #1364

Merged
merged 2 commits into from
Feb 3, 2018
Merged

KAFKA-3949: Avoid race condition when subscription changes during rebalance #1364

merged 2 commits into from
Feb 3, 2018

Conversation

dpkp
Copy link
Owner

@dpkp dpkp commented Feb 1, 2018

Fix #1241

jeffwidman
jeffwidman approved these changes Feb 1, 2018
View reviewed changes
Copy link
Collaborator

@jeffwidman jeffwidman left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, this is a lot more work than I remembered when I investigated how Java fixed this... is that because you're also porting in other updates from the Java client? Or is my memory just faulty?

kafka/coordinator/consumer.py Outdated
@@ -119,6 +121,7 @@ def __init__(self, client, subscription, metrics, **configs):
self.consumer_sensors = ConsumerCoordinatorMetrics(
metrics, self.config['metric_group_prefix'], self._subscription)

#self._handle_metadata_update(self._cluster)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add this if it's commented out?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will delete -- thanks

kafka/coordinator/consumer.py
# work that out right now. If you read this at some later date after the mutable
# state has bitten you... I'm sorry! It mimics the java client, and that's the
# best I've got for now.
self._joined_subscription = set(self._subscription.subscription)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😃 loved this comment

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I know. Hate it, as the method is basically a getter. They put it there, just cause it should be in ConsumerCoordinator but not in AbstractCoordinator and only this method was overridable... Basically a big hack. BTW the method is called metadata() nowdays. https://github.com/apache/kafka/blob/1.0/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L148

@dpkp
Copy link
Owner Author

dpkp commented Feb 2, 2018

Wow, this is a lot more work than I remembered when I investigated how Java fixed this... is that because you're also porting in other updates from the Java client? Or is my memory just faulty?

This is just 3949. The upstream PR is here: apache/kafka#1762

@dpkp dpkp force-pushed the KAFKA_3949_subscription_metadata_race branch from 9ae71f0 to 6ca8439 Compare February 2, 2018 23:26
@dpkp dpkp merged commit 618c505 into master Feb 3, 2018
@jeffwidman jeffwidman deleted the KAFKA_3949_subscription_metadata_race branch February 6, 2018 01:48
@dpkp dpkp mentioned this pull request Dec 29, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tvoinarovskyi tvoinarovskyi tvoinarovskyi approved these changes

@jeffwidman jeffwidman jeffwidman approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dpkp @jeffwidman @tvoinarovskyi