fedup: Parallelize package downloads with "async" #8
Conversation
Normal "yum update" does this -- there's no reason fedup should not, when it deals with even larger sets of packages. This could help the "prepare to upgrade" stage quite a bit. Testing without this patch shows single connection at a time behavior; with patch, we fetch from multiple mirrors. (iftop.) Note that the UI is a little bit broken -- we lose the progress bar. Somehow yum manages to keep it in the face of async downloads, so I suspect it's possible to fix. But I'd rather have parallel downloads than a progress bar any day ;-).
|
Note, for origin/legal purposes: This is my original (though trivial) work, and you can do whatever you want with it under the MIT or GPLv2 license (or probably anything else, I think MIT should cover most everything). |
Modify our on-failure callback (which is called *every* time a package fails, not after N retries) to permit retrying. URLGrabber will blacklist the server after some number of failures, and Fedup will finish downloading even in the face of some bad servers. Set 'retries' on RepoConfig (grandparent of FedupDownloader) to five.
|
The second patch makes fedup resilient to transient server errors and/or bad mirrors. With both patches my average network speed (as measured by iftop) is roughly line speed, despite fetching a ton of tiny packages (F18->F19 texlive pkgs). Like the first patch, this is my original work, released under the same liberal licensing blah blah. |
|
|
|
@tomspur What's your question? If my changes are released under MIT you're perfectly welcome to incorporate them in a GPLv2+ source base like fedup. To be clear: my changes being under MIT does not in any way cause |
|
I doubt that you are allowed to choose another licence as the original one from See https://github.com/wgwoods/fedup/blob/master/fedup/download.py#L8 here for the "or later versions". |
|
@tomspur My changes are my original work. They're pretty useless on their own, but on their own I see no reason why they can't be MIT licensed. Once you apply them to the GPLv2+ work that is fedup, of course, the collective work is GPLv2+. And yes, they're pretty pointless without being applied on top of fedup... Still, is there an actual issue here? Perhaps something technically wrong? One can bikeshed about licenses all day, but I'd rather just make fedup better... |
|
I've added async downloading (with the proper progress bar!) in 7dcbade. The retry handling will be added once I figure out how yum handles that internally. |
|
Awesome. Thanks @wgwoods! |
|
Okay! commit 2d2c958 should fix the failover/retry problems - we were getting in our own way. |
Normal "yum update" does this -- there's no reason fedup should not,
when it deals with even larger sets of packages. This could help the
"prepare to upgrade" stage quite a bit.
Testing without this patch shows single connection at a time behavior;
with patch, we fetch from multiple mirrors. (iftop.)
Note that the UI is a little bit broken -- we lose the progress bar.
Somehow yum manages to keep it in the face of async downloads, so I
suspect it's possible to fix. But I'd rather have parallel downloads
than a progress bar any day ;-).