AttributeError: 'NoneType' object has no attribute '_abort' on aiohttp connection close

[pquentin]

Two separate users of Rally stumbled upon the same aiohttp/asyncio exception.

  File "/.../rally/esrally/driver/driver.py", line 1783, in run
    await es.close()
  File "/.../.local/lib/python3.11/site-packages/elasticsearch/_async/client/__init__.py", line 557, in close
    await self.transport.close()
  File "/.../.local/lib/python3.11/site-packages/elastic_transport/_async_transport.py", line 386, in close
    await node.close()
  File "/.../.local/lib/python3.11/site-packages/elastic_transport/_node/_http_aiohttp.py", line 242, in close
    await self.session.close()
  File "/.../.local/lib/python3.11/site-packages/aiohttp/client.py", line 980, in close
    await self._connector.close()
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.local/lib/python3.11/site-packages/aiohttp/connector.py", line 792, in close
    return super().close()
           ^^^^^^^^^^^^^^^
  File "/.../.local/lib/python3.11/site-packages/aiohttp/connector.py", line 409, in close
    self._close()
  File "/.../.local/lib/python3.11/site-packages/aiohttp/connector.py", line 439, in _close
    transport.abort()
  File "/.../.pyenv/versions/3.11.2/lib/python3.11/asyncio/sslproto.py", line 247, in abort
    self._ssl_protocol._abort()
    ^^^^^^^^^^^^^^^^^^^^^^^^^

AttributeError: 'NoneType' object has no attribute '_abort'

This error has been triggered by python/cpython#98540 which fixed python/cpython#98539. It is in Python 3.11.1 and not Python 3.11.0. Since downgrading to Python 3.11.0 in #1712 we have not observed it.

@danielmitterdorfer analyzed the problem and concluded that the problem is that aiohttp attempts to close the transport several times and only checks for None in two places:

1. https://github.com/aio-libs/aiohttp/blob/2be9318dc95a26940e9095673ebcbbf77dfd087b/aiohttp/connector.py#L357-L358
2. https://github.com/aio-libs/aiohttp/blob/2be9318dc95a26940e9095673ebcbbf77dfd087b/aiohttp/connector.py#L411-L413

They should guard this with if transport is not None and not transport.is_closing() instead.

We should report the bug to aiohttp, ideally with a reproducer, and then upgrade our aiohttp and Python version.

[Dreamsorcerer]

Done: aio-libs/aiohttp#7280

But, if you do have time to figure out a test that can reproduce it, it'd be great to have that regression test.

[Dreamsorcerer]

Actually, I think in updating the existing tests to pass, this is probably covered sufficiently.

[pquentin]

Thanks!

So this was a Python bug, that will be fixed in Python 3.11.4: python/cpython#104485. It should be released on Monday, 2023-06-05 according to https://peps.python.org/pep-0664/. At that point we should update .ci/variables.json from 3.11.0 to 3.11.4.