Skip to content

Conversation

@jsancio
Copy link
Member

@jsancio jsancio commented Feb 9, 2021

Implement Raft Snapshot loading API.

  1. Adds a new method handleSnapshot to raft.Listener which is called whenever the RaftClient determines that the Listener needs to load a new snapshot before reading the log. This happens when the Listener's next offset is less than the log start offset also known as the earliest snapshot.

  2. Adds a new type SnapshotReader<T> which provides a Iterator<Batch<T>> interface and de-serializes records in the RawSnapshotReader into Ts

  3. Adds a new type RecordsIterator<T> that implements an Iterator<Batch<T>> by scanning a Records object and deserializes the batches and records into Batch<T>. This type is used by both SnapshotReader<T> and RecordsBatchReader<T> internally to implement the Iterator interface that they expose.

  4. Changes the MockLog implementation to read one batch at a time. The previous implementation always read from the given offset to the high-watermark. This made it impossible to test interesting snapshot loading scenarios.

  5. Removed throws IOException from some methods. Some of types were inconsistently throwing IOException in some cases and throwing RuntimeException(..., new IOException(...)) in others. This PR improves the consistent by wrapping IOException in RuntimeException in a few more places and replacing Closeable with AutoCloseable.

  6. Updated the Kafka Raft simulation test to take into account snapshot. ReplicatedCounter was updated to generate snapshot after 10 records get committed. This means that the ConsistentCommittedData validation was extended to take snapshots into account.

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.

Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@jsancio
Copy link
Member Author

jsancio commented Mar 29, 2021

Update the PR's description to match the latest changes included here.

Copy link
Member

@mumrah mumrah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch! First pass of questions/comments inline

@jsancio jsancio force-pushed the kafka-12154-snapshot-loading-api branch from 10cd065 to 029fea5 Compare April 10, 2021 04:33
@jsancio
Copy link
Member Author

jsancio commented Apr 10, 2021

@mumrah @hachikuji @dengziming thanks for the feedback. I should have addressed all of your suggestions. Let me know if I missed anything.

Copy link
Contributor

@hachikuji hachikuji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the patch!

@hachikuji hachikuji merged commit 6203bf8 into apache:trunk May 1, 2021
@jsancio jsancio deleted the kafka-12154-snapshot-loading-api branch May 1, 2021 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants