Escaping of double quotes in query string is not consistent with requests

[Ch00k]

It seems to me that escaping of the URL query string is not entirely correct. Here is an example with a real URL:

In [47]: httpx.get('https://www.songkick.com/api/universal_search?query="foo"')
Out[47]: <Response [400 Bad Request]>

In [48]: httpx.get('https://www.songkick.com/api/universal_search?query="foo"').url
Out[48]: URL('https://www.songkick.com/api/universal_search?query="foo"')

In [49]: requests.get('https://www.songkick.com/api/universal_search?query="foo"')
Out[49]: <Response [200]>

In [50]: requests.get('https://www.songkick.com/api/universal_search?query="foo"').url
Out[50]: 'https://www.songkick.com/api/universal_search?query=%22foo%22'

I dug into the code, and found that the root cause is the WHATWG_SAFE variable here: we consider " as one of the safe characters, so it would never get percent-escaped. This was introduced in #3188, which refers to https://github.com/web-platform-tests/wpt/blob/master/url/resources/urltestdata.json. This test data file seems to have a case for a double quotes in query string, and the case seems to show that the double quote is percent-escaped.

I was going to open a PR, but decided to hold off until I have better understanding where does the value of WHATWG_SAFE come from, and whether the double quote there is intentional or a mistake.

[tomchristie]

@Ch00k Thanks for that.

Starting point here would be to paste each of those URLs into chrome, and inspect which characters are percent escaped there. Perhaps try other browsers too and see how consistent they are.

Both styles are okay, but I believe we're more in line with browser behavior.

It might also be worth reviewing https://url.spec.whatwg.org/#percent-encoded-bytes, in particular the "The query percent-encode set is the ...", "The path percent-encode set is the..." etc. - double check that against our escape sets and make sure we're linking with appropriate comments in that section of the codebase.

I noticed that the test case you linked to has the " character in the fragment (#anchor) section, not the query section.

Upshot...

• We should be commenting in the codebase linking to the appropriate parts of WHATWG.
• It is different to requests, which at least deserves a comment in the codebase.