Installing LuaJIT sometimes fails with SHA256 mismatch

[mpeterv]

For instance, when installing LuaJIT 2.0.4 this can happen:

Fetching LuaJIT from https://github.com/LuaJIT/LuaJIT/archive/v2.0.4.tar.gz
Verifying SHA256 checksum
Error: SHA256 checksum mismatch for /home/mpeterv/.cache/hererocks/LuaJIT-2.0.4.tar.gz
Expected: d2abdf16bd3556c41c0aaedad76b6c227ca667be8350111d037a4c54fd43abad
Observed: 324f191dcf771310be0fc6496f7282387be9cf8d38b105566226c6cb079e90eb

Happens with LuaJIT 2.1.0-beta2, too. SHA256 for other downloads seemingly don't mismatch. Observed invalid checksums are the same every time.

Comparing normal and invalid files (at least for LuaJIT 2.0.4) shows that the invalid one is almost correct but has extra 7 bytes prepended and appended to it. Prefix is d0a4f\r\n and suffix is \r\n0\r\n\r\n.

The failures happen on Travis/Appveyor fairly often but I've only managed to reproduce it locally once.

[mpeterv]

Reproduced locally for LuaJIT 2.1.0, again, get the expected file with prefix and suffix, prefix is e8119\r\n and suffix is the same, \r\n0\r\n\r\n. Reproduction done using urllib.urlretrieve, which is what hererocks calls.

[mpeterv]

Calling urlretrieve in a loop shows that the first download for each url is broken, the following ones are OK. Perhaps either github or urlretrieve handle chunked encoding incorrectly?..

[mpeterv]

Using curl --verbose, indeed, GitHub appears to use Transfer-Encoding: chunked, but only on the first request for each url, with a reset after some delay.

[mpeterv]

Of course, the "prefix" is file length in hexadecimal, so the broken file is just full chunk encoded file. Not sure why GitHub decides to send the whole file in one chunk, or why urlretrieve can't handle it,

[mpeterv]

OK, so urllib doesn't support chunked transfer (httplib has functions for working with it but they are not used), but urllib2 does, and so does urllib.request from Python 3.

[mpeterv]

Fixed in @a9b8ab9.