Tuple index out of range with 3.7.1 and no IPv6 #5156

danielnelson · 2020-10-26T22:09:12Z

🐞 Describe the bug
After upgrading to aiohttp==3.7.1, dns resolution fails with an exception.

💡 To Reproduce

I believe this is due to not have IPV6 enabled in my kernel, which is custom compiled with.
CONFIG_IPV6=n

Run:

import asyncio
import aiohttp


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://python.org') as response:
            print(response.status)


asyncio.run(main())

Actual behavior

$ python bug.py
Traceback (most recent call last):
  File "bug.py", line 11, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "bug.py", line 7, in main
    async with session.get('http://python.org') as response:
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 1124, in __aenter__
    self._resp = await self._coro
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 528, in _request
    req, traces=traces, timeout=real_timeout
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 541, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 898, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 1005, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 871, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/resolver.py", line 37, in resolve
    if family == socket.AF_INET6 and address[3]:  # type: ignore
IndexError: tuple index out of range

💡 Expected behavior

Should print 200.

📋 Logs/tracebacks

📋 Your version of the Python

$ python --version
Python 3.7.9

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires: multidict, async-timeout, typing-extensions, yarl, chardet, attrs
Required-by: replay-proxy

$ python -m pip show multidict
Name: multidict
Version: 4.7.6
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires:
Required-by: yarl, aiohttp

$ python -m pip show yarl
Name: yarl
Version: 1.5.1
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires: typing-extensions, multidict, idna
Required-by: aiohttp, replay-proxy

📋 Additional context

The text was updated successfully, but these errors were encountered:

asvetlov · 2020-10-27T09:55:05Z

I'm curious what parameters are passed into getaddrinfo() in your case and what list is returned?
Could you print them in a comment?

My initial thought is: on systems with IPv6 not installed the check if family == socket.AF_INET6 and address[3]: should fail on family == socket.AF_INET6 test; adress[3] test is not executed in this case.

danielnelson · 2020-10-28T00:24:39Z

I added some debug statements to resolver.py:

  async def resolve(
      self, hostname: str, port: int = 0, family: int = socket.AF_INET
  ) -> List[Dict[str, Any]]:
+     print('getaddrinfo(%s, %s, type=%s, family=%s)' % (hostname, port, type, family))
      infos = await self._loop.getaddrinfo(
          hostname, port, type=socket.SOCK_STREAM, family=family
      )
+     print('infos:', infos)

getaddrinfo(python.org, 80, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('45.55.99.72', 80))]
getaddrinfo(python.org, 443, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('45.55.99.72', 443))]
getaddrinfo(www.python.org, 443, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00'))]
Traceback (most recent call last):
  File "bug-5156.py", line 11, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "bug-5156.py", line 7, in main
    async with session.get('http://python.org') as response:
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 1124, in __aenter__
    self._resp = await self._coro
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 528, in _request
    req, traces=traces, timeout=real_timeout
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 541, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 898, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 1005, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 871, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/resolver.py", line 39, in resolve
    if family == socket.AF_INET6 and address[3]:  # type: ignore
IndexError: tuple index out of range

For reference, here is my current DNS resolution for www.python.org

$ host www.python.org
www.python.org is an alias for dualstack.python.map.fastly.net.
dualstack.python.map.fastly.net has address 151.101.188.223
dualstack.python.map.fastly.net has IPv6 address 2a04:4e42:2d::223

asvetlov · 2020-10-28T13:42:39Z

Interesting...
In (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') the (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') doesn't look like IPv6 at all!
Is it correct?
I mean if I do host, port = (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') -- both host and port are broken anyway.

danielnelson · 2020-10-28T18:38:15Z

I found this which explains the cause: https://bugs.python.org/issue16208

FWIW, I was able to process the results of getaddrinfo in C. Here is some embarrassingly bad C code:

#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#include <arpa/inet.h>
#include <stdlib.h>


#define BUF_SIZE 500

void
print_addr(struct addrinfo *result) {
	char *addr = NULL;
	switch(result->ai_family) {
		case AF_INET: {
			struct sockaddr_in *addr_in = (struct sockaddr_in *)result->ai_addr;
			addr = malloc(INET_ADDRSTRLEN);
			inet_ntop(AF_INET, &(addr_in->sin_addr), addr, INET_ADDRSTRLEN);
			break;
		}
		case AF_INET6: {
			struct sockaddr_in6 *addr_in6 = (struct sockaddr_in6 *)result->ai_addr;
			addr = malloc(INET6_ADDRSTRLEN);
			inet_ntop(AF_INET6, &(addr_in6->sin6_addr), addr, INET6_ADDRSTRLEN);
			break;
		}
		default:
			break;
	}
	printf("IP address: %s\n", addr);
	free(addr);
}

int
main(int argc, char *argv[])
{
	struct addrinfo hints;
	struct addrinfo *result, *rp;
	int sfd, s, j;
	size_t len;
	ssize_t nread;
	char buf[BUF_SIZE];

	if (argc < 3) {
		fprintf(stderr, "Usage: %s host port msg...\n", argv[0]);
		exit(EXIT_FAILURE);
	}

	memset(&hints, 0, sizeof(struct addrinfo));
	hints.ai_family = AF_UNSPEC;    /* Allow IPv4 or IPv6 */
	hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */
	hints.ai_flags = 0;
	hints.ai_protocol = 0;          /* Any protocol */

	s = getaddrinfo(argv[1], argv[2], &hints, &result);
	if (s != 0) {
		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s));
		exit(EXIT_FAILURE);
	}

	while (result != 0) {
		print_addr(result);
		result = result->ai_next;
	}
}

$ gcc bug.c
$ ./a.out www.python.org 443
IP address: 151.101.188.223
IP address: 2a04:4e42:2d::223

I tried to do something similar in Python, I'm not sure if I'm doing it right though:

import socket

for ai in socket.getaddrinfo('www.python.org', 443):
    print(ai)
    if ai[0] == socket.AF_INET6:
        af, socktype, proto, canonname, sa = ai
        print("AF_INET6", socket.inet_ntop(socket.AF_INET6, sa[1]))

$ python getaddrinfo.py 
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443))
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('1imagine51.101.188.223', 443))
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('151.101.188.223', 443))
(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00'))
Traceback (most recent call last):
  File "getaddrinfo.py", line 7, in <module>
    print("AF_INET6", socket.inet_ntop(socket.AF_INET6, sa[1]))
ValueError: unknown address family 10

I'm running a Gentoo system with the -ipv6 use flag set, but I enabled it for the Python build as it seems problematic.

Perhaps a fix in aiohttp would be to check at AF_INET6 addresses are a 4 tuple and ignore if they are not.

Also, I think this is the same issue as #4557

webknjaz · 2020-10-28T18:46:38Z

I'm running a Gentoo system with the -ipv6 use flag set, but I enabled it for the Python build as it seems problematic.

So it's enabled in CPython but disabled in the kernel? What about sysctl? Could you post your sudo sysctl -a | grep ipv6 output? Does it have something like net.ipv6.conf.lo.disable_ipv6 set? How about other interfaces? Or is this setting just missing?

webknjaz · 2020-10-28T18:50:41Z

Also, I think this is the same issue as #4557

So it should probably be mitigated with "has ipv6" check like https://github.com/tornadoweb/tornado/pull/670/files#diff-e9701aba4f8165d329018bef200b0d98dd70c24856c159b5c29cce89ddd69e8bR56

asvetlov · 2020-10-28T19:53:18Z

Perhaps a fix in aiohttp would be to check at AF_INET6 addresses are a 4 tuple and ignore if they are not.

From a practical point of view, I'm ok with this solution if it raises a warning before the ignoring.
From my understanding, we see a perfect example of a gray corner case that we can pass around easily.
A detection for address tuple size smells bad but looks working.

danielnelson · 2020-10-28T21:15:10Z

So it's enabled in CPython but disabled in the kernel? What about sysctl?

Sorry, my comment was a little confusing, when I reported this issue it was disabled in Python and in the Kernel. My workaround is to enable it in Python and leave it disabled in the Kernel.

Could you post your sudo sysctl -a | grep ipv6 output?

It looks like there are no results containing ipv6.

I like the socket.has_ipv6 solution, maybe we should test for this when creating the TCPConnector and set the family to AF_UNSPEC or AF_INET. Essentially the linked tornado code.

webknjaz · 2020-10-28T23:49:12Z

So I looked at those bugs on CPython tracker and I'm puzzled why they are "wontfix". Moreover, I think that CPython may be incorrectly processing NUL-terminated strings or something like that.

I've tried decoding that bytesting manually on a Gentoo laptop with IPv6 on and still got an error (a different error but still):

>>> socket.inet_ntop(socket.AF_INET6, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid length of packed IP address string

Something about it looked weird and I tried to do the packing of a known address 2a04:4e42:41::223:

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('2a04:4e42:41::223', 443, 0, 0))]
>>> socket.inet_pton(socket.AF_INET6, '2a04:4e42:2d::223')
b'*\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#'

Woah! That looks different! Let's try unpacking that correct value, just to be sure:

>>> socket.inet_ntop(socket.AF_INET6, b'*\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#')
'2a04:4e42:2d::223'

Yaaay! We've got the correct value!

Based on this, I'm pretty much convinced that the bug in CPython originates in socket.getaddrinfo() returning a malformed packed IPv6 address.

P.S. It's interesting that both representations have identical *\x04NB\x00-\x00\x00 substring ("\x01\xbb\x00\x00\x00\x00 *\x04NB\x00-\x00\x00" and " *\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#") so I'm convinced it has something to do with processing the memory offsets on the C level.

webknjaz · 2020-10-29T00:04:45Z

So I looked up the getaddrinfo(3) manpage and it seems like the fix could be as easy as adding a socket.AI_ADDRCONFIG as a flag for the getaddrinfo() call:

If hints.ai_flags includes the AI_ADDRCONFIG flag, then IPv4
addresses are returned in the list pointed to by res only if the
local system has at least one IPv4 address configured, and IPv6
addresses are returned only if the local system has at least one IPv6
address configured. The loopback address is not considered for this
case as valid as a configured address. This flag is useful on, for
example, IPv4-only systems, to ensure that getaddrinfo() does not
return IPv6 socket addresses that would always fail in connect(2) or
bind(2).

@danielnelson could you please try this in your Python REPL:

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)

and also (just in case):

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)

webknjaz · 2020-10-29T00:21:25Z

Other observations:

The broken bytesting is 14 bytes long while the correct one is 16 bytes
I haven't noticed anything immediately obvious in https://github.com/python/cpython/blob/master/Modules/socketmodule.c so maybe this is how the underlying C-libs behave...

danielnelson · 2020-10-29T01:32:08Z

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443))]

asvetlov · 2020-10-29T08:15:55Z

socket.has_ipv6 is compile-time constant. I wonder how can it help.
@danielnelson do you have has_ipv6 == False on your system?

webknjaz · 2020-10-29T09:34:31Z

@asvetlov has_ipv6 sounds like a band-aid. It'll probably help but flags=socket.AI_ADDRCONFIG is preferable as it'll make getaddrinfo() return only the address families that the network stack on the machine actually supports.

asvetlov · 2020-10-29T10:36:51Z

On Linux, flags=0 for getaddrinfo() is considered as (AI_V4MAPPED | AI_ADDRCONFIG) according to docs: https://man7.org/linux/man-pages/man3/getaddrinfo.3.html

I read it as AI_ADDRCONFIG is passed.

Anyway, whatever we decide we need a PR that passes at least both our CI and works well on @danielnelson box.

Does anybody want to create it?

danielnelson · 2020-10-29T18:26:33Z

I believe AI_ADDRCONFIG is not passed by default because Python always passed a hints struct.

If I call s = getaddrinfo(argv[1], argv[2], NULL, &result);only IPv4 addresses are returned. But if I call with a zeroed hints struct I get all addresses families:

memset(&hints, 0, sizeof(struct addrinfo));
s = getaddrinfo(argv[1], argv[2], &hints, &result);

webknjaz · 2020-10-30T19:03:40Z

@asvetlov It's what the underlying C API defaults to.

But no, it not the default in the CPython stdlib. Both the source (https://github.com/python/cpython/blob/dff1ad5/Modules/socketmodule.c#L6463) and the docs (https://docs.python.org/3/library/socket.html#socket.getaddrinfo) confirm that flags=0.

So I'm still convinced that setting AI_ADDRCONFIG is the best solution here.

This reverts commit 9d81913.

This reverts commit 48de143.

…k supports (#5176) * Fix for #5156 * test for #5156 * add changes file * rearrange if/else * Revert "rearrange if/else" This reverts commit a557e4c. * Revert "test for #5156" This reverts commit 9d81913. * Revert "Fix for #5156" This reverts commit 48de143. * Add AI_ADDRCONFIG flag to loop.getaddrinfo * update changes file

…k supports (#5176) (#5190) * Fix for #5156 * test for #5156 * add changes file * rearrange if/else * Revert "rearrange if/else" This reverts commit a557e4c. * Revert "test for #5156" This reverts commit 9d81913. * Revert "Fix for #5156" This reverts commit 48de143. * Add AI_ADDRCONFIG flag to loop.getaddrinfo * update changes file Co-authored-by: Dmitry Erlikh <derlih@gmail.com>

…k supports (#5176) (#5189) * Fix for #5156 * test for #5156 * add changes file * rearrange if/else * Revert "rearrange if/else" This reverts commit a557e4c. * Revert "test for #5156" This reverts commit 9d81913. * Revert "Fix for #5156" This reverts commit 48de143. * Add AI_ADDRCONFIG flag to loop.getaddrinfo * update changes file Co-authored-by: Dmitry Erlikh <derlih@gmail.com>

…k supports (aio-libs#5176) * Fix for aio-libs#5156 * test for aio-libs#5156 * add changes file * rearrange if/else * Revert "rearrange if/else" This reverts commit a557e4c. * Revert "test for aio-libs#5156" This reverts commit 9d81913. * Revert "Fix for aio-libs#5156" This reverts commit 48de143. * Add AI_ADDRCONFIG flag to loop.getaddrinfo * update changes file

webknjaz · 2021-07-19T20:47:50Z

A new issue #5901 just came in today reporting this problem on the recent version of the 3.7 stream (3.7.4.post0). So I'm reopening this for now. I think it may be reasonable to come up with a docker image based on Gentoo with the ipv6 USE-flag disabled globally and enabled just for CPython. We could then use that image to verify the fix and also make a regression test using it.

dilyanpalauzov · 2021-07-19T21:05:48Z

As mentioned at #5901 the 3.8 branch fails on my system. Here is the output of the questions above:

import socket
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x03\x00\x00'))]

>>> socket.has_ipv6 == False
True

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.12.223', 443)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x03\x00\x00'))]

I have libc 2.33 and kernel 4.14.217

The patch from #5901 is included in the tip of the 3.8 branch, but does not help.

Hanaasagi · 2021-07-20T04:56:22Z

Something that might be helpful. Use a self-compiled CPython with --enable-ipv6 option. And disable disable ipv6 via sysctl -w net.ipv6.conf.all.disable_ipv6=1. Then call socket.getaddrinfo with family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG, will get a socket.gaierror: [Errno -2] Name or service not known.

Python 3.8.5 (default, Jul 20 2021, 04:42:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.has_ipv6
True
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/Python-3.8.5/Lib/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.108.223', 443))]

Update:
This can reproduce with

Enable ipv6 stack in Linux: sysctl -w net.ipv6.conf.all.disable_ipv6=0
Compile CPython with --disable-ipv6 option

Python 3.8.5 (default, Jul 20 2021, 05:11:52)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.has_ipv6
False
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x1a\x00\x00'))]

See the CPython implemention makesockaddr fucntion:

If we don't enable ipv6 when compile CPython, this case is missing.

#ifdef ENABLE_IPV6
    case AF_INET6:
    {
        const struct sockaddr_in6 *a = (const struct sockaddr_in6 *)addr;
        PyObject *addrobj = make_ipv6_addr(a);
        PyObject *ret = NULL;
        if (addrobj) {
            ret = Py_BuildValue("OiII",
                                addrobj,
                                ntohs(a->sin6_port),
                                ntohl(a->sin6_flowinfo),
                                a->sin6_scope_id);
            Py_DECREF(addrobj);
        }
        return ret;
    }

https://github.com/python/cpython/blob/7766b96ab80b04509bbac708ee5ecf3c1c5934fc/Modules/socketmodule.c#L1237-L1252

And it will go to the default case. Just return the sa_family and raw bytes.

    default:
        /* If we don't know the address family, don't raise an
           exception -- return it as an (int, bytes) tuple. */
        return Py_BuildValue("iy#",
                             addr->sa_family,
                             addr->sa_data,
                             sizeof(addr->sa_data));

https://github.com/python/cpython/blob/7766b96ab80b04509bbac708ee5ecf3c1c5934fc/Modules/socketmodule.c#L1436-L1445

aiohttp will get a integer 10, but it doesn't mean a AddressFamily.AF_INET6. Unfortunately, it's a IntEnum, so we will mistake this for a ipv6 address.

>>> socket.AddressFamily.AF_INET6 == 10
True

My suggestion is to use socket.has_ipv6 for pre-check.

dilyanpalauzov · 2021-07-20T08:32:27Z

I have compiled python with ./configure --enable-loadable-sqlite-extensions --disable-ipv6 --with-system-expat --with-system-libmpdec --enable-shared.

Hanaasagi · 2021-07-21T16:46:12Z

Link a fix: #5906.

asvetlov · 2021-10-24T12:06:32Z

Fixed

See #916 (comment) See aio-libs/aiohttp#5156

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.7.0 to 3.7.4. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/aio-libs/aiohttp/releases">aiohttp's releases</a>.</em></p> <blockquote> <h2>aiohttp 3.7.3 release</h2> <h2>Features</h2> <ul> <li>Use Brotli instead of brotlipy <code>[#3803](aio-libs/aiohttp#3803) <https://github.com/aio-libs/aiohttp/issues/3803></code>_</li> <li>Made exceptions pickleable. Also changed the repr of some exceptions. <code>[#4077](aio-libs/aiohttp#4077) <https://github.com/aio-libs/aiohttp/issues/4077></code>_</li> </ul> <h2>Bugfixes</h2> <ul> <li>Raise a ClientResponseError instead of an AssertionError for a blank HTTP Reason Phrase. <code>[#3532](aio-libs/aiohttp#3532) <https://github.com/aio-libs/aiohttp/issues/3532></code>_</li> <li>Fix <code>web_middlewares.normalize_path_middleware</code> behavior for patch without slash. <code>[#3669](aio-libs/aiohttp#3669) <https://github.com/aio-libs/aiohttp/issues/3669></code>_</li> <li>Fix overshadowing of overlapped sub-applications prefixes. <code>[#3701](aio-libs/aiohttp#3701) <https://github.com/aio-libs/aiohttp/issues/3701></code>_</li> <li>Make <code>BaseConnector.close()</code> a coroutine and wait until the client closes all connections. Drop deprecated "with Connector():" syntax. <code>[#3736](aio-libs/aiohttp#3736) <https://github.com/aio-libs/aiohttp/issues/3736></code>_</li> <li>Reset the <code>sock_read</code> timeout each time data is received for a <code>aiohttp.client</code> response. <code>[#3808](aio-libs/aiohttp#3808) <https://github.com/aio-libs/aiohttp/issues/3808></code>_</li> <li>Fixed type annotation for add_view method of UrlDispatcher to accept any subclass of View <code>[#3880](aio-libs/aiohttp#3880) <https://github.com/aio-libs/aiohttp/issues/3880></code>_</li> <li>Fixed querying the address families from DNS that the current host supports. <code>[#5156](aio-libs/aiohttp#5156) <https://github.com/aio-libs/aiohttp/issues/5156></code>_</li> <li>Change return type of MultipartReader.<strong>aiter</strong>() and BodyPartReader.<strong>aiter</strong>() to AsyncIterator. <code>[#5163](aio-libs/aiohttp#5163) <https://github.com/aio-libs/aiohttp/issues/5163></code>_</li> <li>Provide x86 Windows wheels. <code>[#5230](aio-libs/aiohttp#5230) <https://github.com/aio-libs/aiohttp/issues/5230></code>_</li> </ul> <h2>Improved Documentation</h2> <ul> <li>Add documentation for <code>aiohttp.web.FileResponse</code>. <code>[#3958](aio-libs/aiohttp#3958) <https://github.com/aio-libs/aiohttp/issues/3958></code>_</li> <li>Removed deprecation warning in tracing example docs <code>[#3964](aio-libs/aiohttp#3964) <https://github.com/aio-libs/aiohttp/issues/3964></code>_</li> <li>Fixed wrong "Usage" docstring of <code>aiohttp.client.request</code>. <code>[#4603](aio-libs/aiohttp#4603) <https://github.com/aio-libs/aiohttp/issues/4603></code>_</li> <li>Add aiohttp-pydantic to third party libraries <code>[#5228](aio-libs/aiohttp#5228) <https://github.com/aio-libs/aiohttp/issues/5228></code>_</li> </ul> <h2>Misc</h2>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst">aiohttp's changelog</a>.</em></p> <blockquote> <h1>3.7.4 (2021-02-25)</h1> <h2>Bugfixes</h2> <ul> <li> <p><strong>(SECURITY BUG)</strong> Started preventing open redirects in the <code>aiohttp.web.normalize_path_middleware</code> middleware. For more details, see <a href="https://github.com/aio-libs/aiohttp/security/advisories/GHSA-v6wp-4m6f-gcjg">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-v6wp-4m6f-gcjg</a>.</p> <p>Thanks to <code>Beast Glatisant <https://github.com/g147></code>__ for finding the first instance of this issue and <code>Jelmer Vernooĳ <https://jelmer.uk/></code>__ for reporting and tracking it down in aiohttp. <code>[#5497](aio-libs/aiohttp#5497) <https://github.com/aio-libs/aiohttp/issues/5497></code>_</p> </li> <li> <p>Fix interpretation difference of the pure-Python and the Cython-based HTTP parsers construct a <code>yarl.URL</code> object for HTTP request-target.</p> <p>Before this fix, the Python parser would turn the URI's absolute-path for <code>//some-path</code> into <code>/</code> while the Cython code preserved it as <code>//some-path</code>. Now, both do the latter. <code>[#5498](aio-libs/aiohttp#5498) <https://github.com/aio-libs/aiohttp/issues/5498></code>_</p> </li> </ul> <hr /> <h1>3.7.3 (2020-11-18)</h1> <h2>Features</h2> <ul> <li>Use Brotli instead of brotlipy <code>[#3803](aio-libs/aiohttp#3803) <https://github.com/aio-libs/aiohttp/issues/3803></code>_</li> <li>Made exceptions pickleable. Also changed the repr of some exceptions. <code>[#4077](aio-libs/aiohttp#4077) <https://github.com/aio-libs/aiohttp/issues/4077></code>_</li> </ul> <h2>Bugfixes</h2> <ul> <li>Raise a ClientResponseError instead of an AssertionError for a blank HTTP Reason Phrase. <code>[#3532](aio-libs/aiohttp#3532) <https://github.com/aio-libs/aiohttp/issues/3532></code>_</li> <li>Fix <code>web_middlewares.normalize_path_middleware</code> behavior for patch without slash. <code>[#3669](aio-libs/aiohttp#3669) <https://github.com/aio-libs/aiohttp/issues/3669></code>_</li> <li>Fix overshadowing of overlapped sub-applications prefixes. <code>[#3701](aio-libs/aiohttp#3701) <https://github.com/aio-libs/aiohttp/issues/3701></code>_</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aio-libs/aiohttp/commit/0a26acc1de9e1b0244456b7881ec16ba8bb64fc3"><code>0a26acc</code></a> Bump aiohttp to v3.7.4 for a security release</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/021c416c18392a111225bc7326063dc4a99a5138"><code>021c416</code></a> Merge branch 'GHSA-v6wp-4m6f-gcjg' into master</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/4ed7c25b537f71c6245bb74d6b20e5867db243ab"><code>4ed7c25</code></a> Bump chardet from 3.0.4 to 4.0.0 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5333">#5333</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/b61f0fdffc887df24244ba7bdfe8567c580240ff"><code>b61f0fd</code></a> Fix how pure-Python HTTP parser interprets <code>//</code></li> <li><a href="https://github.com/aio-libs/aiohttp/commit/5c1efbc32c46820250bd25440bb7ea96cb05abe9"><code>5c1efbc</code></a> Bump pre-commit from 2.9.2 to 2.9.3 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5322">#5322</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/007507580137efcc0a20391a0792f39b337d9c1a"><code>0075075</code></a> Bump pygments from 2.7.2 to 2.7.3 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5318">#5318</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/5085173d947e6cc01b6daf1aa48fe7698834c569"><code>5085173</code></a> Bump multidict from 5.0.2 to 5.1.0 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5308">#5308</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/5d1a75e68d278c641c90021409f4eb5de1810e5e"><code>5d1a75e</code></a> Bump pre-commit from 2.9.0 to 2.9.2 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5290">#5290</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/6724d0e7a944fd7e3a710dc292d785fa8fe424fd"><code>6724d0e</code></a> Bump pre-commit from 2.8.2 to 2.9.0 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5273">#5273</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/c688451ce31b914c71b11d2ac6c326b0c87e6d1f"><code>c688451</code></a> Removed duplicate timeout parameter in ClientSession reference docs. (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5262">#5262</a>) ...</li> <li>Additional commits viewable in <a href="https://github.com/aio-libs/aiohttp/compare/v3.7.0...v3.7.4">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aiohttp&package-manager=pip&previous-version=3.7.0&new-version=3.7.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/neondatabase/neon/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Vadim Kharitonov <vadim@neon.tech>

danielnelson added the bug label Oct 26, 2020

webknjaz added the need pull request label Oct 29, 2020

derlih added a commit to derlih/aiohttp that referenced this issue Oct 29, 2020

Fix for aio-libs#5156

681afd6

derlih added a commit to derlih/aiohttp that referenced this issue Oct 29, 2020

test for aio-libs#5156

04c2f30

derlih added a commit to derlih/aiohttp that referenced this issue Oct 29, 2020

Fix for aio-libs#5156

48de143

derlih added a commit to derlih/aiohttp that referenced this issue Oct 29, 2020

test for aio-libs#5156

9d81913

derlih mentioned this issue Oct 29, 2020

Only query host address families over DNS that the local network stack supports #5176

Merged

5 tasks

derlih added a commit to derlih/aiohttp that referenced this issue Oct 31, 2020

Revert "test for aio-libs#5156"

10057e8

This reverts commit 9d81913.

derlih added a commit to derlih/aiohttp that referenced this issue Oct 31, 2020

Revert "Fix for aio-libs#5156"

eb15898

This reverts commit 48de143.

webknjaz added regression Something that used to work stopped working "as before" after upgrade reproducer: present This PR or issue contains code, which reproduce the problem described or clearly understandable STR labels Oct 31, 2020

webknjaz closed this as completed in #5176 Nov 1, 2020

bdraco mentioned this issue Jan 2, 2021

Nest integration failing due to aiohttp Network unreachable home-assistant/core#44719

Closed

jarp0l mentioned this issue Feb 26, 2021

Bump aiohttp from 3.6.2 to 3.7.4 jarp0l/flafla-bot#1

Merged

webknjaz mentioned this issue Jul 19, 2021

resolver.py/ThreadResolver: address[3] does not exist #5901

Closed

1 task

webknjaz reopened this Jul 19, 2021

WhyNotHugo mentioned this issue Jul 20, 2021

The aiohttp change with storage/dav/http pimutils/vdirsyncer#916

Closed

Hanaasagi mentioned this issue Jul 21, 2021

Check if ipv6 is enabled, when processing socket address info #5906

Merged

5 tasks

asvetlov closed this as completed Oct 24, 2021

WhyNotHugo pushed a commit to pimutils/vdirsyncer that referenced this issue Nov 2, 2021

Require aiohttp >= 3.8.0

961203e

See #916 (comment) See aio-libs/aiohttp#5156

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tuple index out of range with 3.7.1 and no IPv6 #5156

Tuple index out of range with 3.7.1 and no IPv6 #5156

danielnelson commented Oct 26, 2020 •

edited by webknjaz

Loading

asvetlov commented Oct 27, 2020

danielnelson commented Oct 28, 2020

asvetlov commented Oct 28, 2020

danielnelson commented Oct 28, 2020

webknjaz commented Oct 28, 2020

webknjaz commented Oct 28, 2020

asvetlov commented Oct 28, 2020

danielnelson commented Oct 28, 2020

webknjaz commented Oct 28, 2020

webknjaz commented Oct 29, 2020 •

edited

Loading

webknjaz commented Oct 29, 2020

danielnelson commented Oct 29, 2020 •

edited by webknjaz

Loading

asvetlov commented Oct 29, 2020

webknjaz commented Oct 29, 2020

asvetlov commented Oct 29, 2020

danielnelson commented Oct 29, 2020 •

edited by webknjaz

Loading

webknjaz commented Oct 30, 2020

webknjaz commented Jul 19, 2021

dilyanpalauzov commented Jul 19, 2021

Hanaasagi commented Jul 20, 2021 •

edited

Loading

dilyanpalauzov commented Jul 20, 2021

Hanaasagi commented Jul 21, 2021

asvetlov commented Oct 24, 2021

Tuple index out of range with 3.7.1 and no IPv6 #5156

Tuple index out of range with 3.7.1 and no IPv6 #5156

Comments

danielnelson commented Oct 26, 2020 • edited by webknjaz Loading

asvetlov commented Oct 27, 2020

danielnelson commented Oct 28, 2020

asvetlov commented Oct 28, 2020

danielnelson commented Oct 28, 2020

webknjaz commented Oct 28, 2020

webknjaz commented Oct 28, 2020

asvetlov commented Oct 28, 2020

danielnelson commented Oct 28, 2020

webknjaz commented Oct 28, 2020

webknjaz commented Oct 29, 2020 • edited Loading

webknjaz commented Oct 29, 2020

danielnelson commented Oct 29, 2020 • edited by webknjaz Loading

asvetlov commented Oct 29, 2020

webknjaz commented Oct 29, 2020

asvetlov commented Oct 29, 2020

danielnelson commented Oct 29, 2020 • edited by webknjaz Loading

webknjaz commented Oct 30, 2020

webknjaz commented Jul 19, 2021

dilyanpalauzov commented Jul 19, 2021

Hanaasagi commented Jul 20, 2021 • edited Loading

dilyanpalauzov commented Jul 20, 2021

Hanaasagi commented Jul 21, 2021

asvetlov commented Oct 24, 2021

danielnelson commented Oct 26, 2020 •

edited by webknjaz

Loading

webknjaz commented Oct 29, 2020 •

edited

Loading

danielnelson commented Oct 29, 2020 •

edited by webknjaz

Loading

danielnelson commented Oct 29, 2020 •

edited by webknjaz

Loading

Hanaasagi commented Jul 20, 2021 •

edited

Loading