Exception on URL with encoded emoji

[ElliotGarbus]

For URLs that have an emoji I am seeing the httpx.InvalidURL exception. The same url works fine in requests.
Is there a way to get these urls to work in httpx, or is this an httpx bug?

Here is the test case:

from urllib.parse import urlparse, urlunparse
import requests
import httpx

def convert_idna_address(url: str) -> str:
    parsed_url = urlparse(url)
    return urlunparse(
        parsed_url._replace(netloc=parsed_url.netloc.encode("idna").decode("ascii"))
    )


url = 'https://🍺🌯.to/api/v1/instance/peers'
converted_url = convert_idna_address(url)
print(f'{url=} {converted_url=}')

r = requests.get(converted_url)  # this works...
out = r.json()
print(f'requests: {out}')

r = httpx.get(converted_url)  # this does not work, InvalidURL exception
out = r.json()
print(f'httpx: {out}')

Here is the output:

url='https://🍺🌯.to/api/v1/instance/peers' converted_url='https://xn--sh8hve.to/api/v1/instance/peers'
requests: ['gnomos.org', 'toots.dgplug.org', 'social.sdf.org', 'mastodon.aether.us', ...]  # abbreviated here
Traceback (most recent call last):
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\EG\url_test.py", line 20, in <module>
    r = httpx.get(url)    # this does not
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_api.py", line 189, in get
    return request(
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_api.py", line 100, in request
    return client.request(
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_client.py", line 808, in request
    request = self.build_request(
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_client.py", line 348, in build_request
    url = self._merge_url(url)
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_client.py", line 378, in _merge_url
    merge_url = URL(url)
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_urls.py", line 82, in __init__
    raise InvalidURL(message=str(exc)) from None
httpx.InvalidURL: The authority (🍺🌯.to) is not valid.

Process finished with exit code 1

Edited to add version information
httpx version: 0.23.1
Python version: 3.11
OS: WIndows11

Same issues also seen:
httpx version: 0.23.3
Python version: 3.9.12
OS: WIndows11

[ElliotGarbus]

It looks like the code that is causing the error has been updated in the master branch.
Is there a way to pip install httpx directly from github?

[euri10]

It looks like the code that is causing the error has been updated in the master branch. Is there a way to pip install httpx directly from github?

should be pip install git+https://github.com/encode/httpx.git

[ElliotGarbus]

Thanks, I'll give it a try, and reply back with the results.

[ElliotGarbus]

This updated code did NOT fix the issue.

I did the following:
pip uninstall httpx
pip install git+https://github.com/encode/httpx.git@master

pip show Successfully installed httpx-0.23.3

There error is now in a different spot... check_label in idna\core.py

url='https://🍺🌯.to/api/v1/instance/peers' converted_url='https://xn--sh8hve.to/api/v1/instance/peers'
requests: ['gnomos.org', 'toots.dgplug.org', 'social.sdf.org', 'mastodon.aether.us', ..., 'wandering.shop'] # edited
Traceback (most recent call last):
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\EG\url_test.py", line 20, in <module>
    r = httpx.get(converted_url)  # this does not work, InvalidURL exception
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_api.py", line 189, in get
    return request(
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_api.py", line 100, in request
    return client.request(
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_client.py", line 801, in request
    request = self.build_request(
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_client.py", line 358, in build_request
    return Request(
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_models.py", line 350, in __init__
    self._prepare(headers)
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_models.py", line 385, in _prepare
    if not has_host and self.url.host:
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\httpx\_urls.py", line 186, in host
    host = idna.decode(host)
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\idna\core.py", line 390, in decode
    s = ulabel(label)
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\idna\core.py", line 304, in ulabel
    check_label(label)
  File "C:\Users\ellio\PycharmProjects\KivyHelp210\venv\lib\site-packages\idna\core.py", line 250, in check_label
    raise InvalidCodepoint('Codepoint {} at position {} of {} not allowed'.format(_unot(cp_value), pos+1, repr(label)))
idna.core.InvalidCodepoint: Codepoint U+1F37A at position 1 of '🍺🌯' not allowed

[arihantaraza]

I encounter the same issue calling any url having, or redirecting to domains that are not compatible to idna 2008.

Code snippet:
r= httpx.get('http://zenbeauty.site', follow_redirects=True)
or

r= httpx.get('https://xn--melscandles-qi3f.square.site', follow_redirects=True)

Traceback
Traceback (most recent call last): File "/snap/pycharm-community/387/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode coro = func() File "<input>", line 1, in <module> File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_api.py", line 198, in get return request( File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_api.py", line 106, in request return client.request( File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_client.py", line 827, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_client.py", line 914, in send response = self._send_handling_auth( File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth response = self._send_handling_redirects( File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_client.py", line 999, in _send_handling_redirects raise exc File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_client.py", line 988, in _send_handling_redirects request = self._build_redirect_request(request, response) File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_client.py", line 460, in _build_redirect_request url = self._redirect_url(request, response) File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_client.py", line 511, in _redirect_url if url.scheme and not url.host: File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/httpx/_urls.py", line 189, in host host = idna.decode(host) File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/idna/core.py", line 388, in decode s = ulabel(label) File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/idna/core.py", line 299, in ulabel check_label(label) File "/home/arihanta/PycharmProjects/pythonProject1/.venv/lib/python3.10/site-packages/idna/core.py", line 250, in check_label raise InvalidCodepoint('Codepoint {} at position {} of {} not allowed'.format(_unot(cp_value), pos+1, repr(label))) idna.core.InvalidCodepoint: Codepoint U+2019 at position 2 of 'l’shops' not allowed

Configuration:
httpx==0.27.0
python: 3.10
ubuntu 22

Can httpx.URL handle decoding idna 2003 domains (

httpx/httpx/_urls.py

Line 191 in db9072f

host = idna.decode(host)

) ?

    if host.startswith("xn--"):
        try:
            host = idna.decode(host)
        except idna.InvalidCodepoint:
            host = host.encode("utf-8").decode("idna")

[tomchristie]

This is a good reminder thanks. I'd suggest that we possibly don't want to perform IDNA decoding.

Comparison here...

• Chrome displays the URL without decoding https://xn--lshops-3h0c.myshopify.com/
• Safari displays the URL with decoding https://l’shops.myshopify.com

Since the host is so important in terms of identity, I'd tend towards chrome-like API behaviours in this case.

Need to expand on this more coherantly, but voicing this as a starting point.

[arihantaraza]

In that case, would handling the idna.InvalidCodepoint error or not trying to decode domains that are not compatible to the idna module be a coherent behaviour ?

[jruere]

Accepting already encoded domains would work, but IDNA2003 encoded domains will need to be encoded eventually, as DNS requires it.

By accepting encoded domains, the /policy/ about what to do in these conditions is passed to the application. I think this is a sane approach.

Should an issue be created?