refactor file downloading

[JeffBezanson]

There are now >=3 implementations of download: one in Base, one in WinRPM, and one in BinDeps. They are all slightly different in unhelpful ways. This is bananas.

Everything we know about how to download files should go in Base.download. That should support trying multiple methods, retrying, timeouts, etc. We need two basic versions, one that downloads to a file, and one that returns all the data (or a stream). BinDeps has a version that returns a command to run. That needs to go away, and it should instead return a closure that calls download.

see e.g. JuliaPackaging/WinRPM.jl#40

[jiahao]

also: JuliaLang/julia#8148

[jiahao]

I wrote my own basic downloader in World of Julia because I got tired of the shelling out overhead from download and the inability to suppress output to stderr. Perhaps some can take that code and flesh it out into a full-featured downloader.

[JeffBezanson]

I think for Base.download we'll have to focus on robust but possibly slow approaches.

Unless perhaps "download for package management purposes" is a distinct operation that should be owned by BinDeps. However people probably don't want to remove download from base.

[StefanKarpinski]

How slow can shelling out possibly be compared to downloading something from the Internet?

[StefanKarpinski]

Also note that implementing downloading functionality in Julia requires the Requests package at least. Not sure we want all of that in Base Julia, although maybe it's fine, but that's what's entailed.

[quinnj]

I think long-term it'd be great to have pure-Julia Requests functionality, but there's a bit of a ways to go (native TLS, HttpParser, etc.). There have been some active discussions lately: https://github.com/JuliaWeb/Roadmap/issues

[JeffBezanson]

Yes, that's out of scope for this issue. The point is that even within the world of downloaders that use external programs, there is too much fragmentation and duplication.

[jiahao]

@StefanKarpinski I think Requests reuses live connections. It's definitely a lot faster for the World of Julia use case of downloading many small files. But since this is out of scope I won't say any more.