4.4.0rc2: read() called while another coroutine is already waiting for incoming data

[JonasKs]

Version: 4.4.0rc2

Platform: MacOS and Windows. Python 3.10.6.

Description: We've noticed that we are no longer able to enqueue_job() with arq. All of this works in 4.4.0rc1.

Traceback (most recent call last):
  File "<redacted>", line <redacted>, in <redacted>
    await redis.enqueue_job(
  File "<redacted>", line 137, in enqueue_job
    if any(await asyncio.gather(pipe.exists(job_key), pipe.exists(result_key_prefix + job_id))):
  File "<redacted>/lib/python3.10/site-packages/redis/asyncio/client.py", line 1161, in immediate_execute_command
    return await conn.retry.call_with_retry(
  File "<redacted>lib/python3.10/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
    return await do()
  File "<redacted>lib/python3.10/site-packages/redis/asyncio/client.py", line 463, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
  File "<redacted>lib/python3.10/site-packages/redis/asyncio/client.py", line 1293, in parse_response
    result = await super().parse_response(connection, command_name, **options)
  File "/<redacted>/lib/python3.10/site-packages/redis/asyncio/client.py", line 505, in parse_response
    response = await connection.read_response()
  File "<redacted>/lib/python3.10/site-packages/redis/asyncio/connection.py", line 886, in read_response
    response = await self._parser.read_response(
  File "<redacted>/lib/python3.10/site-packages/redis/asyncio/connection.py", line 468, in read_response
    await self.read_from_socket()
  File "<redacted>/lib/python3.10/site-packages/redis/asyncio/connection.py", line 451, in read_from_socket
    buffer = await self._stream.read(self._read_size)
  File "<redacted>/lib/python3.10/asyncio/streams.py", line 669, in read
    await self._wait_for_data('read')
  File "<redacted>lib/python3.10/asyncio/streams.py", line 488, in _wait_for_data
    raise RuntimeError(
RuntimeError: read() called while another coroutine is already waiting for incoming data

I can try to provide a reproducible example later if you'd like, but that would require arq as a dependency.

[chayim]

@dvora-h It looks like #2308 has the unintentional side effect here. I'm going to dig further. But if you take 4.4.0rc1 and layer until here - this is when it drops.

[chayim]

@mainframeindustries or @utkarshgupta137 Can I ask you for a hand here? I've traced down to the the change here (#2308) - combined with the stack trace above.

I think it's related to how arq configures it's connection by default - and a retry should be configured. But honestly, weak in async.

[utkarshgupta137]

@chayim I'm not very familiar with non-cluster parts, but I'll take a look. @JonasKs Can you share a minimum example? From what I can tell, it could be related to running 2 operations on the same pipeline in parallel. I think it is an unsupported (& unsound?) operation. Can you try creating 2 pipelines & then using asyncio.gather?

[JonasKs]

I'm on vacation, so I can't really until a week or two. I think this demo script should work though: https://github.com/samuelcolvin/arq/blob/main/docs/examples/main_demo.py

See docs here

(The example does not configure redis port and uses the default port, but you can change it by using redis_settings)

[samuelcolvin]

Hi all, thanks for reporting this and notifying me.

I'm pretty busy until Wednesday - I can review a PR, but I won't be able to look into this further until the end of the week.

Is there any particular reason not to delay the release until the end of the week?

[chayim]

@chayim I'm not very familiar with non-cluster parts, but I'll take a look. @JonasKs Can you share a minimum example? From what I can tell, it could be related to running 2 operations on the same pipeline in parallel. I think it is an unsupported (& unsound?) operation. Can you try creating 2 pipelines & then using asyncio.gather?

Just to be clear, since I helped someone else with a bug like this earlier, pipelines should be dedicated for collected use. Agreed. FWIW I think it's a usage bug, but I wanted another set of eyes rather than trigger a poor release.

Is there any particular reason not to delay the release until the end of the week?

@samuelcolvin We have a lot of asks (awesome!) from the community, and we'd love to get 4.4.x out the door, since this is the biggest release since 4.1.4, maybe beforehand. But, a 4.3.5 is about to land - we cherry-picked several items, so in my mind this reduces the pressure.

With 4.3.5 coming out soon, I don't feel pressured to release this right away. But, I'd love to close it off, and could definitely use a hand!

[JonasKs]

I believe this can be closed now?

[chayim]

Indeed. IMHO we can close this. We can even release down 4.4.0 now. That'll come shortly. Thanks everyone!

[gerazenobi]

Hi there 👋

I was running the demo example and run into this issue: RuntimeError: read() called while another coroutine is already waiting for incoming data

Versions

arq                0.24.0
redis              4.5.4

[xiefantasy]

你好呀👋

我正在运行演示示例并遇到了这个问题：RuntimeError: read() called while another coroutine is already waiting for incoming data

版本

arq                0.24.0
redis              4.5.4

I had the same problem with this version

[hzliu]

same here, with version 4.5.4

[masc-it]

same

[JonasKs]

Please create a new issue, with your own up to date stack trace and findings. I do no longer experience this, it's impossible for anyone to troubleshoot on the words "same".