Ensure read-lock is not continuously held on a section while iterating over concurrent maps #9787
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…g over concurrent maps
merlimat
added
the
type/enhancement
merlimat
requested review from
sijie,
srkukarni,
rdhabalia,
jerrypeng and
codelipenghui
lhotari
reviewed
Mar 3, 2021
| } finally { | ||
| if (acquiredReadLock) { | ||
| storedKey = (K) table[bucket]; | ||
| storedValue = (V) table[bucket + 1]; | ||
| unlockRead(stamp); |
There was a problem hiding this comment.
try-finally for unlock? (Same comment to all unlock locations)
There was a problem hiding this comment.
Sure, I think in all places here it should be guaranteed to not throw, but makes sense to do it as general practice.
|
Nice! Can't wait to try it out. |
codelipenghui
approved these changes
Mar 4, 2021
|
Sorry. I came too late to the party. +1 |
mlyahmed
pushed a commit
to mlyahmed/pulsar
that referenced
this pull request
Mar 5, 2021
…g over concurrent maps (apache#9787) * Ensure read-lock is not continuously held on a section while iterating over concurrent maps * Added try/finally
|
btw. I happened to come across #8877 which was a fix to a deadlock. |
This was referenced Mar 11, 2021
merlimat
added a commit
that referenced
this pull request
May 22, 2021
…g over concurrent maps (#9787) * Ensure read-lock is not continuously held on a section while iterating over concurrent maps * Added try/finally
eolivelli
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 24, 2021
…g over concurrent maps (apache#9787) * Ensure read-lock is not continuously held on a section while iterating over concurrent maps * Added try/finally (cherry picked from commit 4c369c9)
codelipenghui
pushed a commit
that referenced
this pull request
May 25, 2021
### Motivation In several places in the code when iterating over the custom hashmaps, we are taking over a copy of the map. This was done every time the iteration could end up modifying the map, since there was a non-reentrant mutex taken during the iteration. Any modification would lead to a deadlock. Since the behavior was changed in #9787 to not hold the section mutex during the iteration, there's no more need to make a copy of the maps.
yangl
pushed a commit
to yangl/pulsar
that referenced
this pull request
Jun 23, 2021
### Motivation In several places in the code when iterating over the custom hashmaps, we are taking over a copy of the map. This was done every time the iteration could end up modifying the map, since there was a non-reentrant mutex taken during the iteration. Any modification would lead to a deadlock. Since the behavior was changed in apache#9787 to not hold the section mutex during the iteration, there's no more need to make a copy of the maps.
This was referenced Jul 22, 2021
BewareMyPower
added a commit
to streamnative/kop
that referenced
this pull request
Jul 23, 2021
Fixes #618 ### Motivation See #618 (comment) for the deadlock analysis. ### Modifications - Use `ConcurrentHashMap` instead of `ConcurrentLongHashMap`. Though this bug may already be fixed in apache/pulsar#9787, the `ConcurrentHashMap` from Java standard library is more reliable. The possible performance enhancement brought by `ConcurrentLongHashMap` still needs to be proved. - Use `AtomicBoolean` as `KafkaTopicConsumerManager`'s state instead of read-write lock to avoid `close()` method that tries to acquire write lock blocking. - Run a single cursor expire task instead one task per channel, since #404 changed `consumerTopicManagers` to a static field, there's no reason to run a task for each connection.
BewareMyPower
added a commit
to BewareMyPower/kop
that referenced
this pull request
Jul 25, 2021
Fixes streamnative#618 ### Motivation See streamnative#618 (comment) for the deadlock analysis. ### Modifications - Use `ConcurrentHashMap` instead of `ConcurrentLongHashMap`. Though this bug may already be fixed in apache/pulsar#9787, the `ConcurrentHashMap` from Java standard library is more reliable. The possible performance enhancement brought by `ConcurrentLongHashMap` still needs to be proved. - Use `AtomicBoolean` as `KafkaTopicConsumerManager`'s state instead of read-write lock to avoid `close()` method that tries to acquire write lock blocking. - Run a single cursor expire task instead one task per channel, since streamnative#404 changed `consumerTopicManagers` to a static field, there's no reason to run a task for each connection.
BewareMyPower
added a commit
to streamnative/kop
that referenced
this pull request
Jul 25, 2021
Fixes #618 ### Motivation See #618 (comment) for the deadlock analysis. ### Modifications - Use `ConcurrentHashMap` instead of `ConcurrentLongHashMap`. Though this bug may already be fixed in apache/pulsar#9787, the `ConcurrentHashMap` from Java standard library is more reliable. The possible performance enhancement brought by `ConcurrentLongHashMap` still needs to be proved. - Use `AtomicBoolean` as `KafkaTopicConsumerManager`'s state instead of read-write lock to avoid `close()` method that tries to acquire write lock blocking. - Run a single cursor expire task instead one task per channel, since #404 changed `consumerTopicManagers` to a static field, there's no reason to run a task for each connection.
bharanic-dev
pushed a commit
to bharanic-dev/pulsar
that referenced
this pull request
Mar 18, 2022
### Motivation In several places in the code when iterating over the custom hashmaps, we are taking over a copy of the map. This was done every time the iteration could end up modifying the map, since there was a non-reentrant mutex taken during the iteration. Any modification would lead to a deadlock. Since the behavior was changed in apache#9787 to not hold the section mutex during the iteration, there's no more need to make a copy of the maps.
yws-tracy
reviewed
Jan 3, 2023
There was a problem hiding this comment.
hi mrlimat @merlimat @eolivelli @lhotari :
I do not quite understand enhance for loop is replaced with "for (int i = 0...) " in this mr, as I know for array loop, enhanced for loop is quite equivalent to "for (int i = 0...) ", Is there any other purpose？ look forward to your reply,
thanks a lot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
As discussed in #9764 the fact that we're potentially holding a read-lock while scanning through a section of the map has several implications:
Instead of holding the lock throughout the scan of the section, we should instead release the read lock before calling the processing function, going back into the optimistic read mode.
This will not add any overhead (in terms of volatile reads) compared to the current implementation, but will avoid all the possible deadlock traps, since we're never going to be holding the lock while calling the user code.