fix: avoid acquiring lock on two mutexes at the same time to prevent deadlock #13636
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jessesuen
approved these changes
May 17, 2023
Codecov ReportPatch and project coverage have no change.
Additional details and impacted files@@ Coverage Diff @@
## master #13636 +/- ##
=======================================
Coverage 49.24% 49.24%
=======================================
Files 248 248
Lines 42838 42839 +1
=======================================
+ Hits 21094 21097 +3
+ Misses 19646 19645 -1
+ Partials 2098 2097 -1
☔ View full report in Codecov by Sentry. |
…deadlock Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
alexmt
force-pushed
the
invalidate-settings-lock
branch
from
99cb45f to
cb653f1
Compare
alexmt
added
cherry-pick/2.5
cherry-pick/2.6
cherry-pick/2.7
|
/cherry-pick release-2.7 |
1 similar comment
|
/cherry-pick release-2.7 |
gcp-cherry-pick-bot bot
pushed a commit
that referenced
this pull request
May 18, 2023
…deadlock (#13636) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
|
/cherry-pick release-2.6 |
1 similar comment
|
/cherry-pick release-2.6 |
gcp-cherry-pick-bot bot
pushed a commit
that referenced
this pull request
May 18, 2023
…deadlock (#13636) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
|
/cherry-pick release-2.5 |
gcp-cherry-pick-bot bot
pushed a commit
that referenced
this pull request
May 18, 2023
…deadlock (#13636) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
schakrad
pushed a commit
to schakrad/argo-cd
that referenced
this pull request
Jul 24, 2023
…deadlock (argoproj#13636) (argoproj#13648) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> Signed-off-by: schakrad <58915923+schakrad@users.noreply.github.com>
This was referenced Jul 25, 2023
yyzxw
pushed a commit
to yyzxw/argo-cd
that referenced
this pull request
Aug 9, 2023
…deadlock (argoproj#13636) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #11458
PR fixes an almost identical bug that was fixed by argoproj/gitops-engine#521.
Deadlock is happening because we have two functions that acquire lock on two different mutexes in different order.
Function 1 stack trace:
Function 2 stack trace:
So liveStateCache.invalidate waits for SetPopulateResourceInfoHandler and SetPopulateResourceInfoHandler waits for liveStateCache.invalidate.
PR updates liveStateCache.invalidate to release Mutex 1 before it locks Mutex 2