-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consumer poll loop does not commit offsets in case of application shutdown #557
Comments
I spent some time trying to reproduce the problem you described; but every time the application context is closed, Kafka consumers are terminated in an orderly manner. Log messages confirm that consumers do commit offsets:
They leave the consumer group:
Then close their sessions:
And finally unregister themselves from the cluster before the application is shut down:
Most of these log messages are low-level, so you need to enable them explicitly if you want to read them: <logger name="org.apache.kafka" level="ALL"/> Everything seems to be working as expected, so I'm going to close this issue. Please feel free to raise a new one if it still doesn't work for you. |
Expected Behavior
I am creating a simple
@KafkaListener
which is using defaultenable.auto.commit=true
with 5 second interval (all defaults). When Micronaut application shuts down, the consumer ends abruptly. What I would expect, that the consumer commits already processed offsets synchronously, as it is declared in methodio.micronaut.configuration.kafka.processor.KafkaConsumerProcessor#createConsumerThreadPollLoop
in thecatch
block:However, this code does not seem to executed. I tried increasing
pollTimeout
, but this is probably connected to a fact that consumer thread is just "destroyed" and there's no graceful shutdown. I would also expect, that consumer leaves the consumer group (no logs about that).Actual Behaviour
The consumer "dies" and offsets that were processed are not committed to broker, thus they are re-read when (another) consumer takes the partition.
Steps To Reproduce
micronaut-kafka
Environment Information
Example Application
No response
Version
3.4.0
The text was updated successfully, but these errors were encountered: