Compact definitions during runtime #571
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
carlhoerberg
reviewed
Sep 8, 2023
src/lavinmq/vhost.cr
Outdated
| @@ -19,6 +19,8 @@ module LavinMQ | |||
| include SortableJSON | |||
| include Stats | |||
|
|
|||
| DEFINITIONS_DIRT_COMPACT_THREASHOLD = 10_000 | |||
There was a problem hiding this comment.
let's make it a config variable
There was a problem hiding this comment.
and call it deletes_compacts_definitions? as in "100 'delete' operations compacts the definition file"
src/lavinmq/vhost.cr
Outdated
| # By doing equality comparision we'll only start one fiber | ||
| # without having to keep track if it's started or not | ||
| if @definitions_dirt_counter == DEFINITIONS_DIRT_COMPACT_THREASHOLD | ||
| spawn(name: "VHost #{name} definitions compact") { compact! } |
There was a problem hiding this comment.
why in a separate fiber? makes it potentially thread unsafe
There was a problem hiding this comment.
I think one reason was that store_definition is called from a @definitions_lock.synchronize context, and that compact! also want the lock, and the other was to not block the current fiber with this (even though it's probably fast, and it will block by holding the lock).
With high churns (e.g. tons of shortlived queues) the definitions file can grow indefinitely. This will add compactation every 10000th "delete" frame. (10k was chosen completley arbitrary.)
carlhoerberg
force-pushed
the
compact-definitions-during-runtime
branch
from
d955928 to
9a03fa4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WHAT is this pull request doing?
With high churns (e.g. tons of short-lived queues) the definitions file can grow indefinitely. This will add compaction every 10000th "delete" frame. (10k was chosen completely arbitrary.)
HOW can this pull request be tested?
Run specs, or launch lavinmq and create and delete a queue more than 10k times.