Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix deadlock between pc-control and pc-broker-poll threads where partitions are revoked #548

Merged

Conversation

nachomdo
Copy link
Contributor

@nachomdo nachomdo commented Feb 15, 2023

Fixes #541
image

Checklist

  • Documentation (if applicable)
  • Changelog

@what-the-diff
Copy link

what-the-diff bot commented Feb 15, 2023

  • The copyright year was changed from 2022 to 2023.
  • A new method called maybeAcquireCommitLock() is added in the onPartitionsRevoked(). This will acquire a lock before committing offsets, which prevents multiple threads from accessing this function at the same time and causing race conditions.
  • In PartitionState class, we have removed some unnecessary code that checks if there are any partitions assigned to it or not because now we can be sure that only one thread accesses each partition state object at a given point of time due to locks acquired by maybeAcquireCommitLock(). So no need for synchronization here anymore!
  • We also remove an unused variable "partition" in ProcessingShard class as well as another check condition (if(workContainer != null)) inside WorkContainer's getWork() method since these were used when parallel consumer had more than 1 worker per topic-partition pair but now with just 1 worker per tp pair, they're redundant and cause confusion/bugs so removing them makes things simpler :)

Copy link
Contributor

@rkolesnev rkolesnev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a relevant test.
Please add changelog entry and do a local build or run process-sources to re-generate readme with updated changelog.

@nachomdo nachomdo force-pushed the bugs/fix-rebalance-eos-deadlock branch from c4e92b4 to 455b941 Compare March 10, 2023 14:04
@nachomdo nachomdo requested a review from rkolesnev March 10, 2023 17:58
Copy link
Contributor

@rkolesnev rkolesnev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Transactional PConsumer stuck while rebalancing
3 participants