Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Choose between consumer or producer commit modes #31

Merged
merged 11 commits into from
Nov 23, 2020

Conversation

astubbs
Copy link
Contributor

@astubbs astubbs commented Nov 11, 2020

Fixes #30 - Makes transactions optional
Fixes #25 by locking producer access around a read/write lock, as when using a transactional producer, it's not actually completely thread safe.
Fixes #26 Document config options better - maxConcurrency, numberOfThreads, maxUncommittedMessagesToHandlePerPartition documentation
Fixed #12 Create a parallel consumer which doesn't require a Producer instance enhancement

@astubbs astubbs added the enhancement New feature or request label Nov 11, 2020
@astubbs astubbs marked this pull request as draft November 11, 2020 15:41
@astubbs astubbs force-pushed the features/transactions-optional branch 2 times, most recently from 11802f0 to 42e502c Compare November 11, 2020 17:13
@astubbs astubbs force-pushed the features/transactions-optional branch 4 times, most recently from 5ca0b22 to 893ae25 Compare November 20, 2020 11:33
@astubbs astubbs changed the title Fixes #30 - Makes transactions optional feature: Choose between consumer or producer commit modes Nov 20, 2020
@astubbs astubbs force-pushed the features/transactions-optional branch from 1004717 to 1829f3b Compare November 20, 2020 15:51
@astubbs astubbs marked this pull request as ready for review November 20, 2020 15:52
@astubbs astubbs force-pushed the features/transactions-optional branch 8 times, most recently from f55badb to 52c7819 Compare November 23, 2020 13:32
…mits

- Choose either Consumer sync or async commits
- Fixes confluentinc#25 confluentinc#25:
-- Sometimes a a transaction error occurs - Cannot call send in state COMMITTING_TRANSACTION confluentinc#25
- ReentrantReadWrite lock protects non-thread safe transactional producer from incorrect multithreaded use
- Wider lock to prevent transaction's containing produced messages that they shouldn't
- Implement non transactional synchronous commit sync properly
- Select tests adapted to non transactional as well
- Must start tx in MockProducer as well
- Adds supervision to poller
- Fixes a performance issue with the async committer not being woken up
- Enhances tests to run under multiple commit modes
- Fixes example app tests - incorrectly testing wrong thing and MockProducer not configured to auto complete
- Make committer thread revoke partitions and commit
- Have onPartitionsRevoked be responsible for committing on close, instead of an explicit call to commit by controller
- Make sure Broker Poller now drains properly, committing any waiting work
- Add missing revoke flow to MockConsumer wrapper
- Add missing latch timeout check
…ng (avoid races), add ability to request commit
@astubbs astubbs force-pushed the features/transactions-optional branch from 52c7819 to a90f11c Compare November 23, 2020 15:46
@astubbs astubbs force-pushed the features/transactions-optional branch 2 times, most recently from 398a11f to bcb4f9c Compare November 23, 2020 15:49
@astubbs astubbs force-pushed the features/transactions-optional branch from bcb4f9c to 88261e9 Compare November 23, 2020 15:51
@astubbs astubbs merged commit 00a6a9e into confluentinc:master Nov 23, 2020
@astubbs astubbs deleted the features/transactions-optional branch November 23, 2020 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment