curl instead of URLDownloadToCacheFileW?

[JeffBezanson]

Today I'm in an environment where IE is blocked on windows machines, and as a result this call to download files in WinRPM fails. I saw that curl is included with the git that julia ships on windows, so I tried replacing the download function (https://github.com/JuliaLang/WinRPM.jl/blob/d788882b4a71cd77e85f0d3a61aacb98028881b2/src/WinRPM.jl#L45) with simply

return readall(`curl -L $source`), 200

and everything worked perfectly. Can we incorporate this change somehow? Maybe as an alternative to try if URLDownloadToCacheFileW fails? I can write up a PR.

[tkelman]

curl doesn't work through proxies.

[JeffBezanson]

I believe I'm using a proxy here, and it works for me. I tried unsetting and resetting http_proxy.

[tkelman]

ref JuliaLang/julia#3001 (comment) and JuliaPackaging/BinDeps.jl#121 - does

readall(`powershell -Command "(new-object net.webclient).DownloadString(\"$source\")"`)

work? We won't be bundling git forever.

[JeffBezanson]

Yes, the powershell command also works for me. Excellent.

As long as we have the download function in Base, we might as well consolidate all these downloading methods there.

As an aside, the curl man page says:

curl offers a busload of useful tricks like proxy support,

so I suspect something else might have been the problem in those cases. Seems like we can avoid curl anyway though.

[tkelman]

so I suspect something else might have been the problem in those cases

Quite possible. Or it needs to be configured somehow, or the version of curl that is bundled in Windows git might be old or deficient.

Base.download shells out to whatever it can find on *nix, or uses the IE API in a way that doesn't follow redirects on Windows. The BinDeps.download_cmd is slightly fancier, but not much. Before we lean too much more heavily on powershell we should probably have @ihnorton test some things, he was reporting trouble when starting Julia under mintty. It works for me under Cygwin's mintty though, so really not sure what's up.

[tkelman]

A lot of people have reported download troubles with WinRPM though, so definitely worth trying out some other options to see if we can make it more robust.

[JeffBezanson]

We should move all the functionality to Base (again, unless people don't want the download function in there at all), and teach it every trick in the book, detecting and retrying different methods etc. We should be able to support a monotonically-increasing number of cases that way.

[tkelman]

Pretty sure everyone thinks download is worth having in Base. I agree that it's not ideal to have the Base version be gimpy and incapable of handling many use cases. Should we try a fix here that could help on 0.3 too, or are local workarounds good enough for now and otherwise you're only concerned with 0.4?

[vtjnash]

usually i would point out that it's a really messy to bootstrap this particular package, which is why it has such peculiar implementations in places. but now that we're shipping busybox anyways (http://intgat.tigress.co.uk/rmy/busybox/index.html), we can actually use wget and other standard unix tools

[tkelman]

We have curl from git at the moment, and similarly I don't think wget from busybox would work in strangely configured environments. But it's an okay fallback.

[vtjnash]

https://github.com/rmyorston/busybox-w32/blob/master/networking/wget.c has some support for the http_proxy environment variable

[tkelman]

Another confusing instance of WinRPM download troubles: JuliaIO/LightXML.jl#23

[nalimilan]

The Powershell trick also works for me on a lab machine where download (and WinRPM) fail overwise (#57).

[pfitzseb]

For me as well -- using powershell circumvents IE's Enhanced Security Configuration on Windows Server. I can prepare a PR (against Base?) if there's consensus on using that instead of the current implementation on Windows?

[tkelman]

This package probably needs a pretty big refactoring of the way it does downloading to work well on windows server, use requests instead of httpclient on unix if anyone is using this there, etc. Currently it's passing a status code around in a few places but it's hard to tell whether that's really absolutely necessary.

[stevengj]

Any progress on getting a more full-featured download function in Base? Is there a Base issue for this?

[tkelman]

almost positive there's an issue, but no progress that I know of

[tkelman]

I may have been thinking of JuliaPackaging/BinDeps.jl#135.

There is now a pure julia (except for the TLS library) http parser, but I don't feel comfortable having anything depend on it until it's established and stable