Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(server): correct batch materialization logic in BatchAccumulator #1555

Merged
merged 1 commit into from
Feb 18, 2025

Conversation

hubcio
Copy link
Contributor

@hubcio hubcio commented Feb 18, 2025

This commit refactors the BatchAccumulator by removing the
capacity field and simplifying the logic for materializing
batches. The materialize_batch_and_maybe_update_state method
is renamed to materialize_batch_and_update_state, and the
logic for handling message offsets and timestamps is streamlined.

The changes improve the clarity and efficiency of the batch processing
logic, addressing issues with the previous handling of has_remainder.
Additionally, the logic in writing_messages.rs is updated to reflect
these changes, ensuring that unsaved messages are correctly managed
based on batch size.

Now, instead of having hard-limit of messages_required_to_save we
will trigger it once when capacity if exceeded and save all messages
on disk, instead of handling remainder.

@hubcio hubcio force-pushed the fix-index-handling branch 3 times, most recently from 70a758b to 9afb4b6 Compare February 18, 2025 19:22
@hubcio hubcio force-pushed the fix-index-handling branch from 54d8fc8 to 82a7da5 Compare February 18, 2025 20:29
This commit refactors the `BatchAccumulator` by removing the
`capacity` field and simplifying the logic for materializing
batches. The `materialize_batch_and_maybe_update_state` method
is renamed to `materialize_batch_and_update_state`, and the
logic for handling message offsets and timestamps is streamlined.

The changes improve the clarity and efficiency of the batch processing
logic, addressing issues with the previous handling of `has_remainder`.
Additionally, the logic in `writing_messages.rs` is updated to reflect
these changes, ensuring that unsaved messages are correctly managed
based on batch size.
@hubcio hubcio force-pushed the fix-index-handling branch from 82a7da5 to d926706 Compare February 18, 2025 20:51
@spetz spetz merged commit 5c48f2d into master Feb 18, 2025
14 checks passed
@spetz spetz deleted the fix-index-handling branch February 18, 2025 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants