-
-
Notifications
You must be signed in to change notification settings - Fork 382
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SSH/SCP/SFTP protocol support #267
Conversation
# | ||
# Global storage for SSH connections. | ||
# | ||
_SSH = {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does such global cache play with multiprocessing / forking?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For multiprocessing: no problem, there will be one cache for subprocess.
For forking: I think it will be a problem. We don't want different threads to access the same connection simultaneously. Perhaps we can include the current thread_id in the cache key?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we have any solid contract in place for concurrency (yet).
My naive expectation as a user would be "A 'file' open in one process/thread is always valid for that process/thread, but not necessarily across process or thread boundaries". But we should probably document that explicitly.
How do the other connection types (S3, HTTP, HDFS…) manage in the face of multiple processes or threads?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OTOH they "should be fine". We haven't had any concurrency-related bug reports.
On the other hand, we don't have any explicit testing for concurrency problems, either.
Co-Authored-By: mpenkov <penkov@pm.me>
Continued from #58 cause I couldn't write to the original author's repo.
Fix #36