Feature request: ia download should (optionally?) use range requests to resume downloads (and retries seem broken)

[ethus3h]

According to ia help download, it defaults to retrying 5 times.

These command outputs don't seem to show any retries: https://gist.github.com/ethus3h/b9959869b9b00768cbff1a358efb217a https://gist.github.com/ethus3h/0722f52f5918460e170b5e06628243a8

I'm in a rural area, and have a rather unreliable Internet connection. (It usually goes down every few hours, and can range from dying every couple of minutes when it's bad, to lasting maybe half a day when it's doing well. Often when it goes down, it will be entirely unavailable for anywhere from a few minutes to many hours, as it requires a human to notice and reset the equipment.)

When the connection goes down, ia download does not seem to retry, and will exit with an error status (I use a wrapper shell script to catch this and restart it). ia download does not seem to keep the already-downloaded portion of the file and use a HTTP range request to resume the previous try, either.

Consequently, downloading a moderately large file (e.g. the one in the logs above, ~6GB) usually takes several days and tens or hundreds of GB of bandwidth since it keeps going back to the beginning until it gets lucky and the connection lasts long enough for the file to get through.

This doesn't seem very efficient, so I'm writing this feature request.

Thanks for considering it!

[ethus3h]

Oh, I'm using ia version 1.7.6, by the way.

[jjjake]

@ethus3h I would like to add this functionality to v2.

I'm strongly considering switching from the requests library to pycurl in v2. Resuming partially downloaded files is very easy with pycurl.

Thanks for the feedback!

[baka0815]

What's the status of this?
Resuming of broken downloads would be very nice, as it would also reduce traffic on all sides.

[awsms]

@ethus3h I would like to add this functionality to v2.

I'm strongly considering switching from the requests library to pycurl in v2. Resuming partially downloaded files is very easy with pycurl.

Thanks for the feedback!

What exactly is the v2?

[jjjake]

@awsms that was in reference to version 2 of this library, which has since been shipped (current version is v3.5.0). There are no current plans to switch HTTP libraries, but I still support this feature! I'm not opposed to switching HTTP libraries, but it would be a very big job and would need a very good reason at this point.

[hofmand]

Right now the only way to reliably download a complete item is to rerun the command and check the log until all files download successfully. It would be really good if this could be automated without requiring a wrapper script.