Threadsafety improvements using thread local storage #64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The _opener global object defined in the _opener module was wrapped in a subclass of threading.local to improve mechanize threadsafety. With this change that object is no longer shared between threads and is not required to synchronize the execution of urlopen and urlretrieve. This modified version of mechanize was used successfully as backend for a stress test based on the multi-mechanize framework (*) in which up to 512 agents per user group were active. (*) http://code.google.com/p/multi-mechanize/ Signed-off-by: Dhionel Díaz <ddiaz@cenditel.gob.ve>
|
Thank you for your contribution to mechanize! Following the process in #117, future work on mechanize will be occurring here: Please re-file your PR there (where it will get attention, and hopefully merged) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When mechanize is used in a heavily threaded context, serious contention and threadsafety concerns arise, related to the use of a global _opener object.
The proposed changes place _opener in thread.local storage, eliminating the contention. They has been tested in a scenario where each thread only uses its own mechanize objects, that's enough to support a stress test framework like multi-mechanize (http://testutils.org/multi-mechanize/) in a heavily threaded context. With those changes in the general case, if a thread uses mechanize objects created by another one it would use its local _opener object. If that is ok, there should be no issues.
Complete threadsafety is required for the most general case, and more testing. In the restricted scenario the proposed changes seem to be enough for a stable behaviour.