Fix a race condition in subtable accessor deletion #3460
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The following sequence of events would double-delete the Table: 1. Thread A enters SubtableColumnBase::SubtableMap::refresh_accessor_tree(), holding m_subtable_map_lock. 2. Thread B releases the last reference to a Table in that SubtableMap. 3. Thread B blocks on acquiring m_subtable_map_lock in unbind_ref(). 4. Thread A reaches that Table in the iteration and acquires a strong reference (0 -> 1). 5. Thread A finishes with that Table and releases the reference (1 -> 0). 6. Thread A deletes the Table due to the refcount going to 0. 7. Thread A releases m_subtable_map_lock. 8. Thread B acquires m_subtable_map_lock. 9. Thread B rechecks the (now deleted) Table's refcount and sees 0. 10. Thread B tries to delete the already deleted Table. Fix this by not acquiring and releasing a strong reference to the Table if the refcount is already 0.
|
@tgoyne This is an awesome finding! We've been unable to nail this bug for a long, long time. |
|
This is likely hard to add a unit test for. But can we add a "random" stress test that eventually provokes this? Can that user's test be reduced to do something just in core? |
finnschiermer
approved these changes
Nov 11, 2019
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The following sequence of events would double-delete the Table:
Fix this by not acquiring and releasing a strong reference to the Table if the refcount is already 0.
I couldn't figure out a way to reasonable test this. The repro case supplied by the user just does a bunch of things on a bunch of threads and crashes eventually.