Concurrent clients in a single session raise ServerDisconnectedError: None

[messa]

It looks like the ClientSession connection pool does not handle correctly concurrent requests (but it's just my guess).

Server:

import asyncio
from aiohttp import web

async def index_handler(request):
    await asyncio.sleep(.5)
    return web.Response(text='Test')

app = web.Application()
app.router.add_get('/', index_handler)

web.run_app(app)

Client:

import aiohttp
import asyncio

async def check_link(session):
    async with session.get('http://127.0.0.1:8080/') as response:
        content = await response.text()
        print(repr(content))

async def main():
    while True:
        async with aiohttp.ClientSession() as session:
            asyncio.create_task(check_link(session))
            await asyncio.sleep(.1)

asyncio.run(main())

Output:

(venv) root@d93855937cfe:/xx# python3 simple_server.py  &
[1] 4568
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)

(venv) root@d93855937cfe:/xx# python3 client.py
Task exception was never retrieved
future: <Task finished coro=<check_link() done, defined at client.py:4> exception=ServerDisconnectedError(None)>
Traceback (most recent call last):
  File "client.py", line 5, in check_link
    async with session.get('http://127.0.0.1:8080/') as response:
  File "/xx/venv/lib/python3.7/site-packages/aiohttp/client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "/xx/venv/lib/python3.7/site-packages/aiohttp/client.py", line 497, in _request
    await resp.start(conn)
  File "/xx/venv/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 844, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/xx/venv/lib/python3.7/site-packages/aiohttp/streams.py", line 588, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: None
Task exception was never retrieved

The exception repeats many times.

If remove the await asyncio.sleep(.5) or make it .01 then it works correctly, without any errors.

Your environment

Tested on clean Docker image debian:buster with python3-venv. Same behavior observed also on macOS.

(venv) root@d93855937cfe:/xx# pip freeze
aiohttp==3.5.4
async-timeout==3.0.1
attrs==18.2.0
chardet==3.0.4
idna==2.8
multidict==4.5.2
pkg-resources==0.0.0
yarl==1.3.0

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #975 (aiohttp.ClientSession raises ServerDisconnectedError), #1175 (Unclosed client session warning), #2039 (Graceful shutdown for client sessions), #1686 (Client Session Timeouts - Usage Question), and #957 (Discarding cookies in client session).

[messa]

When I move async with ClientSession() right before the session.get() then it works correctly, but I guess having session-per-request is a kind of antipattern.

import aiohttp
import asyncio

async def check_link():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://127.0.0.1:8080/') as response:
            content = await response.text()
            print(repr(content))

async def main():
    while True:
        asyncio.create_task(check_link())
        await asyncio.sleep(0.1)

asyncio.run(main())

[jettify]

Not sure what is your end goal but this should work:

async def main():
    async with aiohttp.ClientSession() as session:
        while True:
            await check_link(session)
            await asyncio.sleep(.1)

[messa]

@jettify Yes, the variant with “non-overlapping” requests works, but I think the aiohttp client should be able to handle overlapping requests as well. This is just a minimal example to demonstrate the error.

[jettify]

Try something like

    async with aiohttp.ClientSession() as session:
        while True:
            check_link(session)
            await asyncio.sleep(.1)

session should not sit inside loop. Also I suggest to add concurrency control for number active requests.

[asvetlov]

@jettify is right.

Let's return to the problem:

async def main():
    while True:
        async with aiohttp.ClientSession() as session:
            asyncio.create_task(check_link(session))
            await asyncio.sleep(.1)

This code closes the session on exit from async with block before check_link task is finished.
It closes a connection to the server.

Obviously, the code is not correct.

[messa]

I see it. Of course 🤦‍♂️ I don't know why I haven't noticed the wrong placement of async with before. I haven't even noticed that @jettify corrected it in the first answer (I had just focused on the unwanted change from asyncio.create_task() to await).

At least hopefully this issue may help somebody in future to debug similar mistakes :)

Thank you for your work on aiohttp and other libs!

[asvetlov]

you are welcome!

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.