Fix possible deadlock #5963
Merged
Fix possible deadlock #5963
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A write-lock was taken which locks the whole store during a query that needs to expand sources. Under load, writes can start to fail.
This is no longer used in tsm and just peridocially locks everything for no reason now.
|
cc @benbjohnson |
Make it consistent with other conventions where exported funcs take a lock.
|
LGTM 👍 |
|
+1 |
|
I'm late to the game but 👍 too |
|
Cherry-picked to 0.11 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In a load test env, writes started timing out and filling up hinted handoff queues. The traces show hundreds of write goroutines blocked trying to acquire the
tsdb.Storeread lock. There were a few other goroutines stuck trying to acquire thetsdb.Storewrite lock in the periodic maintenance goroutine and during expanding sources during a query.This change removes the maintenance goroutine because it is not used by
tsmengine and just ends up locking the wholetsdb.Storeperiocially. This also switches the write lock taken during expanding sources to a read lock since that code only ever reads fields on thetsdb.Store.