RuntimeError: readuntil()

[iwpnd]

Version: 4.5.0, works in 4.3.5
Platform: Python 3.10 on MacOS

Hi 👋

I'm facing the same issue here with 4.5.0 that was successfully resolved with #2540. While the example of @Tiendil in #2450 works with 4.5.0 now, mine does not - yet works with 4.3.5.

I'm running an instance of Tile38 a service implementing the Redis protocol here and am sending custom commands via send_command.

docker run --rm -p 9851:9851 tile38/tile38:latest

import asyncio
import json
import uuid

from redis.asyncio import Redis


async def main():

    client = Redis(
        host="localhost",
        port=9851,
        db=0,
        username=None,
        password=None,
        retry_on_timeout=True,
        single_connection_client=True,
        client_name="test_client",
    )

    n = 100
    await client.initialize()

    async def problem():
        if client.connection:
            await client.connection.send_command("OUTPUT", "JSON")
            await client.connection.send_command(
                "SET", "FLEET", str(uuid.uuid4()), "POINT", 1, 1
            )
            resp = await client.connection.read_response()
            print(json.loads(resp))

    operations = [problem() for _ in range(n)]

    await asyncio.gather(*operations)

    print("success!")

asyncio.run(main())

It errors with

Traceback (most recent call last):
  File "/Users/foobar/Projects/py/pyle38/tmp/readresponse.py", line 42, in <module>
    asyncio.run(main())
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/foobar/Projects/py/pyle38/tmp/readresponse.py", line 37, in main
    await asyncio.gather(*operations)
  File "/Users/foobar/Projects/py/pyle38/tmp/readresponse.py", line 31, in problem
    response = await client.connection.read_response()
  File "/Users/foobar/Library/Caches/pypoetry/virtualenvs/pyle38-NBBwUiJQ-py3.10/lib/python3.10/site-packages/redis/asyncio/connection.py", line 836, in read_response
    response = await self._parser.read_response(
  File "/Users/foobar/Library/Caches/pypoetry/virtualenvs/pyle38-NBBwUiJQ-py3.10/lib/python3.10/site-packages/redis/asyncio/connection.py", line 256, in read_response
    response = await self._read_response(disable_decoding=disable_decoding)
  File "/Users/foobar/Library/Caches/pypoetry/virtualenvs/pyle38-NBBwUiJQ-py3.10/lib/python3.10/site-packages/redis/asyncio/connection.py", line 266, in _read_response
    raw = await self._readline()
  File "/Users/foobar/Library/Caches/pypoetry/virtualenvs/pyle38-NBBwUiJQ-py3.10/lib/python3.10/site-packages/redis/asyncio/connection.py", line 341, in _readline
    data = await self._stream.readline()
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/streams.py", line 524, in readline
    line = await self.readuntil(sep)
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/streams.py", line 616, in readuntil
    await self._wait_for_data('readuntil')
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/streams.py", line 487, in _wait_for_data
    raise RuntimeError(
RuntimeError: readuntil() called while another coroutine is already waiting for incoming data

Whereas with 4.3.5 I see the expected:

{'ok': True, 'elapsed': '2.666µs'}
{'ok': True, 'elapsed': '56µs'}
{'ok': True, 'elapsed': '917ns'}
{'ok': True, 'elapsed': '10.709µs'}
{'ok': True, 'elapsed': '30.375µs'}

Appreciate your work here!

[Vivanov98]

The reason this doesn't work is because the fix in #2540 synchronises concurrency at the client level, but each individual connection is not coroutine-safe.

If you use client.execute_command instead of client.connection.send_command the above error no longer occurs, i.e.:

await client.execute_command("OUTPUT", "JSON")
resp = await client.execute_command("SET", "FLEET", str(uuid.uuid4()), "POINT", 1, 1)
print(resp)

instead of

await client.connection.send_command("OUTPUT", "JSON")
await client.connection.send_command(
        "SET", "FLEET", str(uuid.uuid4()), "POINT", 1, 1
)
resp = await client.connection.read_response()
print(json.loads(resp))

Does that resolve your issue? Let me know if there are any other issues.

[Tiendil]

If you use client.execute_command instead of client.connection.send_command the above error no longer occurs, i.e.:

Is there documentation about this difference?

If not, it will be good to add some warning about concurrency.

Here, probably: https://redis.readthedocs.io/en/stable/connections.html#redis.connection.Connection.send_command ?

Or rename send_command to _send_command because it is not safe to call it directly from a user code.

[iwpnd]

@Vivanov98, you’re right! Awesome! Thank you for looking into and taking the time to reply. 🙏