Fix NOT_COORDINATOR warnings for Kafka consumer offset commit requests #1693
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the
NOT_COORDINATOR
warning when Kafka consumers try to commit offsets. It's fixed by routing theOffsetCommit
requests to the group coordinator in the KafkaSinkCluster methodroute_requests()
.It also adds a new Kafka integration test case
produce_consume_commit_offsets_partitions1
which callscommit_sync(&offsets)
to commit the offset after a valid consume, and callscommitted
to verify the offset has been successfully committed (refs: Kafka Java driver commitSync(), Kafka Java driver committed(), Kafka C++ driver commit(), Kafka C++ driver committed_offsets()).commit_sync
was chosen overcommit_async
due to the following reasons:We need to check if the offset has been committed after sending the offset commit request, and hence need the
commit_sync
to block until the commit succeeds.It may be possible to wrap
committed()
in the OffsetCommitCallback for thecommitAsync()
method so that the check on committed offsets can be triggered aftercommitAsync()
completes. But it seems complicated to implement and probably not worth the effort.Closes #1687