Try to use .netrc if available and enable session cookies by default

[evetion]

Regarding cookies, see https://curl.se/libcurl/c/CURLOPT_COOKIEFILE.html:

Given an empty or non-existing file or by passing the empty string ("") to this option, you can enable the cookie engine without reading any initial cookies.

Regarding netrc, see https://curl.se/libcurl/c/CURLOPT_NETRC.html:

The use of the ~/.netrc file is optional, and information in the URL is to be preferred. The file will be scanned for the host and user name (to find the password only) or for the host only, to find the first user name and password after that machine, which ever information is not specified.

[StefanKarpinski]

Can you add tests that these options actually work? I'm wondering whether we should enable this by default or allow the user to opt into these features.

[StefanKarpinski]

Bump. This seems like a good change but needs tests.

[evetion]

I'll add tests, still working on the cookie one.

[StefanKarpinski]

Thinking about this, I think we should just do this by default and I'm not even convinced we need an option to turn it off. If someone has bothered to create a .netrc file, presumably they'd want to honor it.

Similarly, if a website sets a cookie during a request, we should serve them back their cookie — we don't have the privacy concerns that browsers do because we would only serve cookies back during the same request, i.e. when there's a redirect from one location to another one.

[evetion]

I'd like to have a control test for this, but it seems there's no way to actually disable the cookie engine once set.

[StefanKarpinski]

What about setopt(easy, CURLOPT_COOKIEFILE, C_NULL)?

[evetion]

Doesn't seem to work, this fails:

# Setup control (disable cookie engine)
downloader = Downloader()
easy_hook = (easy, info) -> begin
    Downloads.Curl.setopt(
        easy,
        Downloads.Curl.CURLOPT_COOKIEFILE, C_NULL)
end
downloader.easy_hook = easy_hook
cookies = download_json(set_cookie_url, downloader=downloader)
@test isempty(cookies)

[StefanKarpinski]

Hmm. That's a shame. I guess we don't strictly need the control. I may file an issue with libcurl.

[StefanKarpinski]

Filed: curl/curl#6889

[evetion]

Nice, that's quite productive. Do you want to wait for a fix (and subsequent release?).

[StefanKarpinski]

No, we should proceed without it since it's a bunch of effort to update the library version (new BinaryBuilder release, bumping version in Julia's makefiles, checksum updates — it's a whole process).

[StefanKarpinski]

I've tried to see if httpbingo has a way to set a cookie and redirect to another domain to see if cross-domain cookie leaks are possible, but I couldn't figure out a way to test that short of writing our own HTTP end point. We don't necessarily need that in the test suite, but I did kind of want to test it to make sure that we can't accidentally leak auth cookies in the case of a redirect. I'm guessing libcurl is too smart to do that, but I wanted to see for myself before turning this feature on for everyone.

[StefanKarpinski]

Nice! Tests passing.

[StefanKarpinski]

This should be a temporary file, shouldn't create files in the current directory.

[StefanKarpinski]

Best practice to explicitly delete it anyway.

[evetion]

I've tried to see if httpbingo has a way to set a cookie and redirect to another domain to see if cross-domain cookie leaks are possible, but I couldn't figure out a way to test that short of writing our own HTTP end point. We don't necessarily need that in the test suite, but I did kind of want to test it to make sure that we can't accidentally leak auth cookies in the case of a redirect. I'm guessing libcurl is too smart to do that, but I wanted to see for myself before turning this feature on for everyone.

Always good to double check these possible security issues. I think this is not possible, as a domain will always be set, either by the website or by curl. For our test the domain is explicitly added.

* Added cookie k1="v1" for domain httpbingo.julialang.org, path /cookies/, expire 0

[StefanKarpinski]

Great work on this.