Make compactRange() non-blocking
#597
Assignees
Labels
enhancement
New feature or request
Kafka-Streams
Issues that are relevant to Kafka Streams - but not only
Comments
udi-speedb
added a commit
that referenced
this issue
Aug 29, 2023
Add a new option to CompactRangeOptions, a callback object. By defaut, the callback is empty so CompactRange() will be blocking (as it is now). If the callback is set, the callback object's callback method will be called upon completion with the completion status.
udi-speedb
added a commit
that referenced
this issue
Aug 29, 2023
Add a new option to CompactRangeOptions, a callback object. By defaut, the callback is empty so CompactRange() will be blocking (as it is now). If the callback is set, the callback object's callback method will be called upon completion with the completion status.
udi-speedb
added a commit
that referenced
this issue
Sep 6, 2023
Add a new option to CompactRangeOptions, a callback object. By defaut, the callback is empty so CompactRange() will be blocking (as it is now). If the callback is set, the callback object's callback method will be called upon completion with the completion status.
Guyme
added
the
Kafka-Streams
Yuval-Ariel
pushed a commit
that referenced
this issue
Sep 18, 2023
Add a new option to CompactRangeOptions, a callback object. By default, the callback is empty so CompactRange() will be blocking (as it is now). If the callback is set, the callback object's callback method will be called upon completion with the completion status.
github-project-automation
bot
moved this from 🔎 Researching
to 📖 Need your Opinion!
in Speedb Roadmap
udi-speedb
added a commit
that referenced
this issue
Sep 21, 2023
Yuval-Ariel
pushed a commit
that referenced
this issue
Sep 21, 2023
|
🎉 |
udi-speedb
added a commit
that referenced
this issue
Nov 23, 2023
Add a new option to CompactRangeOptions, a callback object. By default, the callback is empty so CompactRange() will be blocking (as it is now). If the callback is set, the callback object's callback method will be called upon completion with the completion status.
udi-speedb
added a commit
that referenced
this issue
Nov 23, 2023
udi-speedb
added a commit
that referenced
this issue
Dec 6, 2023
Add a new option to CompactRangeOptions, a callback object. By default, the callback is empty so CompactRange() will be blocking (as it is now). If the callback is set, the callback object's callback method will be called upon completion with the completion status.
udi-speedb
added a commit
that referenced
this issue
Dec 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Owner:
Is your feature request related to a problem? Please describe.
In Kafka Streams up to 2.6, to speed up restoration of RocksDB-backed state stores, we used to apply bulk loading when restoring. We used
Options::prepareForBulkLoad()for that. So all data was first written to L0 during restoration without any compactions. When restoration ended and before normal processing started, we switched the state store back to non-bulk-loading mode. Now compactions started on all the restored data on L0. During a compaction writes to the state store stall. These meant, we sometimes experienced write stalls when normal processing started. The duration of these write stalls exceeded a timeout that we have in Kafka Streams (max.poll.interval.ms for the interested reader) which hindered progress in processing. Due to this issue, we removed bulk loading during restoration.Now we did some improvements in restoration that would maybe allow to re-add bulk loading during restoration. However, we would need to make a manual compaction with
compactRange()before we switch to normal processing for a specific store. Since we have multiple state stores to restore, we do not want to be blocked by the manual compaction of each state store after restoration. We would rather want to start manual compaction for one state store that finished restoration and then continue to restore the other state stores. Once the compaction on the state store has finished we want to switch it to normal processing. For that we need some ways to verify if the compaction has ended like a future.Describe the solution you'd like
A non-blocking
compactRange()and a way to verify that the compaction started withcompactRange()finished.The text was updated successfully, but these errors were encountered: