Client can not handle hostnames with 63 bytes when a port is given in the url

[MirkoDziadzka]

Long story short

The http client does have some problem, when the last part of the hostname becomes too long (but in the valid length of an DNS part)

To reproduce, choose a hostname with 60 bytes in the last part of DNS, add a port and try to access the url:

https://host-12345678901234567890123456789012345678901234567890-name:8888/

Note that the hostname is 60 octets, but hostname with port is 65 octets

Expected behaviour

should work with all valid hostnames (a part can have a length of 63 octets)

Actual behaviour

does not work when the last part of a hostname together with the optional port exceeds 63 octets.

Steps to reproduce

Run the following script. It should print "can not connect". Instead it raises

UnicodeError: label too long
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label too long)
ValueError: URL has an invalid label.

import asyncio
import aiohttp

url = "https://host-12345678901234567890123456789012345678901234567890-name:8888/"
#url = "https://fooooo:8888/"

async def test_bug(session):
    try:
        async with session.get(url) as resp:
            pass
    except aiohttp.errors.ClientOSError:
        print("can not connect")

loop = asyncio.get_event_loop()
with aiohttp.ClientSession(loop=loop) as session:
    loop.run_until_complete(test_bug(session))

Your environment

MacOS 10.9, Python 3.5
using aiohttp-0.21.5

[MirkoDziadzka]

The following patch agains 0.21.5 fixes the problem for me.

diff --git a/build-env/lib/python3.5/site-packages/aiohttp/client_reqrep.py b/build-env/lib/python3.5/site-packages/aiohttp/client_reqrep.py
index 68d0ea5..14e2640 100644
--- a/build-env/lib/python3.5/site-packages/aiohttp/client_reqrep.py
+++ b/build-env/lib/python3.5/site-packages/aiohttp/client_reqrep.py
@@ -117,7 +117,7 @@ class ClientRequest:

         # check domain idna encoding
         try:
-            netloc = netloc.encode('idna').decode('utf-8')
+            # netloc = netloc.encode('idna').decode('utf-8')
             host = host.encode('idna').decode('utf-8')
         except UnicodeError:
             raise ValueError('URL has an invalid label.')

I really do not understand why the netloc is checked here, as it can contain the port. IDNA encoding should probably only be executed on the hostname part without the port.

[asvetlov]

Good catch!
But let me postpone the fix for a while: I want to finish https://github.com/asvetlov/yarl first.
The library should address all these shits with URL encoding, quoting etc.

[asvetlov]

Well, I don't know when yarl will be ready but a fix for this particular case is trivial.
Please don't hesitate to make a Pull Request.

[asvetlov]

Fixed by #1088

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.