Python iRODS Client implementation of parallel transfer #236
Conversation
|
getting there... codacy has some opinions. |
|
Yes , going to work on the
then run the whole test suite and let you know! |
d-w-moore
force-pushed
the
file_desc
branch
3 times, most recently
from
4affcbb
to
2a71778
Compare
|
Tested and ready for merge, unless README is judge to need an update for the parallel transfer. (The demo script and progress bar niceties would be the only possibly compelling reason, as the data_objects manager put/get methods should transparently switch over to parallel mode when sizes are greater than 32MB. ) |
trel
changed the title
d-w-moore
force-pushed
the
file_desc
branch
2 times, most recently
from
d3e95c3
to
d5e5c74
Compare
d-w-moore
force-pushed
the
file_desc
branch
7 times, most recently
from
cd6a76e
to
8a3a892
Compare
|
This is working ideally now, and matches For those who wish to benchmark parallel transfer but are running 4.2.8 release of iRODS server, you can select this branch with which to test, but be aware that it cannot be considered as production quality. For one, acPostProcForPut() is called potentially multiple times for a PUT (once per thread of a transfer). |
d-w-moore
force-pushed
the
file_desc
branch
4 times, most recently
from
b8a9064
to
17e6cdc
Compare
|
there are a couple merge conflicts - after those are cleaned up, we can squash down to clean commits and get every commit to have an issue number. need to release 0.8.6 before we merge this (for what will be 0.9.0) |
|
I had planned on resolving those but was waiting to do everything at once ( test with 4-2-stable, assuming Alan's changes are pretty much all in.) |
|
yeah, these are minor, and i don't expect too much more interaction with the alan server code. |
ok will rebase and test this weekend |
We also add the create argument in the data object open() call which defaults to old behavior, ie letting it default True means we create and open a new replica on the default or otherwise targeted resource (DEST_RESC_NAME_KW if specified) if none already exists.
|
codacy and then we start to squash. |
|
ok going to run tests again also. |
| f.write(self.write_str1.encode('utf-8')) | ||
|
|
||
| ## seek and re-read of just written bytes doesn't work in current 4-2-stable | ||
| ## (see https://github.com/irods/irods/issues/5315) |
There was a problem hiding this comment.
5315 is now closed... so this can now be enabled i think
There was a problem hiding this comment.
Yes , good point - will do that in upcoming testing !
|
Closing in favor of #262 , which includes PRC parallel transfer. |
This pull request addresses a foundational issue #232 ( and a smaller one in #233 ) in the Python iRODS-Client (PRC) and then provides a multithreaded data-transfer implementation for the PRC that is similar (and comparable in performance) to the Parallel Transfer Engine integrated into the iRODS core code. The idea is to manage "N" connections to the server -- see iput and iget -N options -- each connection handling 1/N of the work of the (PUT|GET) operation. Because account credentials are available for reuse, this gets around the server having to encrypt the extra channels independent of, for example, the SSL encrypt capability inherent to the native iRODS connection.