ClientSession stuck after websocket 404

[crusaderky]

Long story short

In a specific combination of aiohttp server and client, it is possible to get the client stuck after a ws_connect() call returns 404 from the server.

Steps to reproduce

In this very specific condition:

• aiohttp server
  AND
• aiohttp client
  AND
• a call to ClientSession.ws_connect() failed with a 404 error (exception caught)
  AND
• the client session context has not been reinitialised afterwards

then a subsequent call to ClientSession.get() will get stuck.

The following things make the error disappear:

• the server does not run aiohttp
• the ClientSession context is exited and re-entered after the failed websocket is initialised

Proof of concept:

import asyncio
import aiohttp
import aiohttp.web
import flask


async def get404(session, url):
    await session.get(url)
    print('pass')

async def ws404(session, url):
    try:
        await session.ws_connect(url)
    except aiohttp.WSServerHandshakeError:
        pass
    print('pass')

async def test1(url):
    print(url + "; session reinitialised after every failure")
    async with aiohttp.ClientSession() as session:
        await get404(session, url)
    async with aiohttp.ClientSession() as session:
        await ws404(session, url)
    async with aiohttp.ClientSession() as session:
        await get404(session, url)

async def test2(url):
    print(url + "; session not reinitialised")
    async with aiohttp.ClientSession() as session:
        await get404(session, url)
        await ws404(session, url)
        await get404(session, url)


async def main():
    # Start flask server
    loop = asyncio.get_event_loop()
    app = flask.Flask('test')
    loop.run_in_executor(None, app.run)

    # Start aiohttp server
    webapp = aiohttp.web.Application()
    runner = aiohttp.web.AppRunner(webapp)
    await runner.setup()
    site = aiohttp.web.TCPSite(runner, port=5001)
    await site.start()

    await asyncio.sleep(.1)

    await test1('http://localhost:5000/notexist')
    await test2('http://localhost:5000/notexist')
    await test1('http://localhost:5001/notexist')
    await test2('http://localhost:5001/notexist')


if __name__ == '__main__':
    asyncio.run(main())

Output:

http://localhost:5000/notexist; session reinitialised after every failure
pass
pass
pass
http://localhost:5000/notexist; session not reinitialised
pass
pass
pass
http://localhost:5001/notexist; session reinitialised after every failure
pass
pass
pass
http://localhost:5001/notexist; session not reinitialised
pass
pass
(stuck)

Stack trace after pressing Ctrl+C shows that the event loop is stuck waiting on select():

Traceback (most recent call last):
  File "/Users/crusaderky/Library/Preferences/PyCharmCE2019.1/scratches/aiohttp_ws_stuck.py", line 57, in <module>
    asyncio.run(main())
  File "/anaconda3/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/anaconda3/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
    self.run_forever()
  File "/anaconda3/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
    self._run_once()
  File "/anaconda3/lib/python3.7/asyncio/base_events.py", line 1739, in _run_once
    event_list = self._selector.select(timeout)
  File "/anaconda3/lib/python3.7/selectors.py", line 558, in select
    kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt

Note: the fact that server and client are served by the same event loop is inconsequential. This issue remains when client and server are running in two separate Python interpreters.

Your environment

MacOS Mojave
anaconda
python 3.7.3-h359304d_0
aiohttp 3.5.4-py37h1de35cc_0

[gjcarneiro]

This is probably the same thing as #3380, actually a server-side problem, not client-side. I have made a PR for that.

[crusaderky]

Closing as duplicate of #3380