download_checkmd5 ContentTooShortError

[bricerebsamen]

Hi

I'm using download_checkmd5.py to download a 200MB file from my server, and it always seems to fail with a ContentTooShortError. My server is quite reliable though: using wget I have no issues at all downloading the file.

I tried using urllib2 but it's the same.

def dlfile(url, dest):
    # Open the url
    f = urllib2.urlopen(url)

    # Open our local file for writing
    with open(dest, "wb") as local_file:
        local_file.write(f.read())

I am now working on a solution using wget...

any idea?

Brice

[tfoote]

Some auto retry logic could be added. Wget defaults to 20 retries. http://www.gnu.org/software/wget/manual/html_node/Download-Options.html

[dirk-thomas]

In order to work reliably the code would need logic to resume a previously aborted download. It could e.g. use PyCurl for this (see http://stackoverflow.com/questions/4533304/python-urlretrieve-limit-rate-and-resume-partial-download).

A pull request for this would be welcome 😄

[wjwwood]

I believe this stackoverflow is talking about this issue and how you might implement this behavior with Python's urllib:

http://stackoverflow.com/questions/6963283/python-urllib2-resume-download-doesnt-work-when-network-reconnects

[bricerebsamen]

@dirk-thomas OK I'm starting on it

[dirk-thomas]

One comment regarding the code @wjwwood referenced: it does not check if the server actually support the range header. It should do that in order not to stitch together wrong parts of the downloaded file.