Add .netrc & cookie support

[evetion]

Netrc files can store logins (username, password) for external websites, so you don't need to provide them in the url when accessing these websites. If used for an central authentication service, reading cookies in the current session is also required.

This setup is pretty common for accessing scientific datasets. For example: https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+cURL+And+Wget

I've made #98 to set these options as default. If this is unwarranted, can we discuss the design to support these as keyword arguments to the download function?

[evetion]

In the meantime, this is the way to do it with the current package, based on some of the tests:

downloader = Downloads.Downloader()
easy_hook = (easy, info) -> begin
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_NETRC, Downloads.Curl.CURL_NETRC_OPTIONAL)
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_COOKIEFILE, "")
end
downloader.easy_hook = easy_hook

download(kwargs...) = Downloads.download(kwargs...;downloader=downloader)

[StefanKarpinski]

Yes, you've discovered that Downloader already supports setting any option that libcurl knows about 😸. I'm reluctant to officially document this since I'm not 100% on the API, primarily because it's not composable: if one bit of code sets an easy_hook and then another bit of code does too, it will clobber the earlier one. Perhaps a way to address that would be to have a stack of easy_hooks and when someone registers a hook it goes on the stack and they all get called in order.

[evetion]

Yeah, I agree this shouldn't be the documented API for end-users. Not only because of the composability, but also because it's quite unfriendly for end users to know about CURL_XXX stuff or (stacks of) easy hooks.

I'd expect a high level API like download(...; compressed=true, netrc=true), following a subset of the curl command line interface arguments. This connects to a lower level API for the mapping from those kwargs on the (stack of) easy hooks for more advanced options. We could call it EasyEasy, Easy2 or ExtraEasy? 🙉

[StefanKarpinski]

I meant a literal stack, like a vector of hooks, not a sequence of hook names. It could be an official "workaround API".

[evetion]

Wouldn't it be nicer to already squash that stack into a Dict like structure so it's clear which options will actually be used?

[evetion]

Closed by #98.