Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deep offset commit resilience in Source #1

Open
JoshuaC215 opened this issue Jan 23, 2019 · 0 comments
Open

Deep offset commit resilience in Source #1

JoshuaC215 opened this issue Jan 23, 2019 · 0 comments

Comments

@JoshuaC215
Copy link
Collaborator

Summary

Instead of using auto-commit (lazy implementation for MVP) which has small potential for data loss in case of application crash, ensure that a message is not committed in the Source Kafka partition until it has been successfully Acked/Failed.

Ideally this should also robustly handle failure to Send()/Fail() (at minimum reporting an Event with sufficient information for logging or similar).

Proposed Timeline

This will be a non-trivial lift since it requires quite a few feature changes and tracking all messages through to use the offset watermark effectively.

Proposed Solutions

For handling timeout / loss in kafka, (e.g. not committing the offset until we guarantee all messages before it have been fully handled) it looks like we would disable automatic commit store, manually track which messages were successfully handled, and then use one of the Commit() methods to manually commit. This is likely a post-MVP feature if we are going to maintain significant use of Kafka for a while. Should be thoughtful to leave sufficient hooks for this in the Receive() and Ack() methods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant