Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Received duplicated records when rebalance occurs #365

Closed
astubbs opened this issue Jul 27, 2022 Discussed in #360 · 0 comments · Fixed by #362
Closed

Received duplicated records when rebalance occurs #365

astubbs opened this issue Jul 27, 2022 Discussed in #360 · 0 comments · Fixed by #362
Assignees
Labels
draft-available verified bug Something isn't working

Comments

@astubbs
Copy link
Contributor

astubbs commented Jul 27, 2022

Discussed in #360

Originally posted by Alessandrovito July 20, 2022

Description of my configuration for parallel consumers:

  • created new JStreamVertxParallelEoSStreamProcessor<>(vertx, wc, options) to permit to invoke vertxHttpWebClientStream to send a http POST request to the endpoint
  • Use Vertx as HTTP engine
  • Set commit mode to CommitMode.PERIODIC_CONSUMER_SYNC and setTimeBetweenCommits to 5 seconds
  • Ordering by ProcessingOrder.KEY

I have created an integration test to push events with specific sequence and during the consuming and sending events to a testing endpoint, is simulated a rebalance process simply connect another kafka consumer with the same groupId and after a while, disconnecting and closing the new kafka consumer.
So when I query my testing endpoint I can observe a lots of data duplication. I would expect to have a persisting strategy in the onPartitionsRevoked callback that commit the current offset map, but looks like the commit occurs only at scheduled time (in my case 5 sec). Could you please confirm this behaviour ? Is it possible to reach Exactly Once Semantics at the http sender level for example waiting that all in-flight events are received by the Vertx and therefore commit the offsets inside the onPartitionsRevoked callback ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
draft-available verified bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant