Error when using Lock from thread in asyncio #773

davidbrochart · 2024-08-28T20:13:33Z

Things to check first

I have searched the existing issues and didn't find my bug already reported there
I have checked that my bug is still present in the latest release

AnyIO version

4.4.0

Python version

3.12.5

What happened?

With the asyncio backend, this code fails with TypeError: cannot create weak reference to 'NoneType' object. It works with the trio backend.

Traceback

Traceback (most recent call last):
  File "/home/david/git/pycrdt/foo.py", line 11, in 
    run(main)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_eventloop.py", line 74, in run
    return async_backend.run(func, args, {}, backend_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2034, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2022, in wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/home/david/git/pycrdt/foo.py", line 9, in main
    await to_thread.run_sync(worker, lock)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/pycrdt/foo.py", line 5, in worker
    from_thread.run_sync(lock.release)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/from_thread.py", line 80, in run_sync
    return async_backend.run_sync_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2239, in run_sync_from_thread
    return f.result()
           ^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2230, in wrapper
    f.set_result(func(*args))
                 ^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_synchronization.py", line 202, in release
    if self._owner_task != get_current_task():
                           ^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_testing.py", line 63, in get_current_task
    return get_async_backend().get_current_task()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2484, in get_current_task
    return AsyncIOTaskInfo(current_task())  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1839, in __init__
    task_state = _task_states.get(task)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/weakref.py", line 452, in get
    return self.data.get(ref(key),default)
                         ^^^^^^^^
TypeError: cannot create weak reference to 'NoneType' object

How can we reproduce the bug?

from anyio import Lock, from_thread, to_thread, run

def worker(lock):
    from_thread.run(lock.acquire)
    from_thread.run_sync(lock.release)

async def main():
    lock = Lock()
    await to_thread.run_sync(worker, lock)

run(main)

The text was updated successfully, but these errors were encountered:

agronholm · 2024-08-29T14:38:48Z

The immediate cause is that it's trying to create a weak reference to None. It's trying to do this because current_task() returns None, but I'm still trying to figure out why it would do that.

davidbrochart · 2024-08-29T14:56:35Z

BTW with #761 the error is different: RuntimeError: The current task is not holding this lock.

Traceback

Traceback (most recent call last):
  File "/home/david/git/anyio/foo.py", line 11, in 
    run(main)
  File "/home/david/git/anyio/src/anyio/_core/_eventloop.py", line 74, in run
    return async_backend.run(func, args, {}, backend_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2173, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2161, in wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/foo.py", line 9, in main
    await to_thread.run_sync(worker, lock)
  File "/home/david/git/anyio/src/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2326, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 872, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/foo.py", line 5, in worker
    from_thread.run_sync(lock.release)
  File "/home/david/git/anyio/src/anyio/from_thread.py", line 80, in run_sync
    return async_backend.run_sync_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2388, in run_sync_from_thread
    return f.result()
           ^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2379, in wrapper
    f.set_result(func(*args))
                 ^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 1728, in release
    raise RuntimeError("The current task is not holding this lock")
RuntimeError: The current task is not holding this lock

smurfix · 2024-08-29T15:04:37Z

RuntimeError: The current task is not holding this lock.

That's to be expected.

The workaround is to use a nice 'async with lock:` block, and call back to the thread from within that.

davidbrochart · 2024-08-29T15:12:24Z

Oh I see, thanks. I guess I should close this issue then, and not give Alex more work than needed.

agronholm · 2024-08-29T18:13:08Z

I agree with @smurfix 's suggestion, but I would still like to understand why there was an error, and why it only happens with asyncio, as the effects of the problem may not be limited to locking.

agronholm · 2024-08-29T18:52:06Z

For posterity, the reason for this was that I had opted to use call_soon_threadsafe() as the backing implementation for run_sync_from_thread(). As such, the code runs in an environment where there is no backing task, hence returning None from current_task(). I'm reopening this as this may cause trouble further down the line, but it's not a priority.

samskiter · 2024-09-26T07:30:19Z

I see this is reopened - I think we've just run onto something similar on 4.6 (previously using 4.4). I haven't dug into it much but googling the error brought me straight to this ticket

samskiter · 2024-09-26T07:47:47Z

Ok have confirmed the same issue on 4.5 and 4.6 of Anyio. Sample trace here:

time=2024-09-26T08:24:38 level=ERROR message="Child process (HeatPumpSetter) failed with exception (cannot create weak reference to 'NoneType' object) 1727335478.2475069" file=supervisor.py line=105
  + Exception Group Traceback (most recent call last):
  |   File "blah.py", line 93, in _child_monitor
  |   File "/app/hub/.venv/lib/python3.11/site-packages/trio/_core/_run.py", line 959, in __aexit__
  |     raise combined_error_from_nursery
  | ExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "blah.py", line 54, in run
    |   File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
    |     return await anext(self.gen)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "blah.py", line 48, in process_context
    |   File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
    |     return await anext(self.gen)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "blah.py", line 104, in controllable_pump
    |   File "/app/hub/.venv/lib/python3.11/site-packages/anyio/_core/_synchronization.py", line 147, in __aenter__
    |     await self.acquire()
    |   File "/app/hub/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1742, in acquire
    |     await AsyncIOBackend.cancel_shielded_checkpoint()
    |   File "/app/hub/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2287, in cancel_shielded_checkpoint
    |     with CancelScope(shield=True):
    |   File "/app/hub/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 400, in __enter__
    |     task_state = _task_states[host_task]
    |                  ~~~~~~~~~~~~^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/weakref.py", line 415, in __getitem__
    |     return self.data[ref(key)]
    |                      ^^^^^^^^
    | TypeError: cannot create weak reference to 'NoneType' object
    +------------------------------------

graingert · 2024-09-26T07:51:45Z

@samskiter oh that's odd, it's using the AsyncIOBackend on trio, can you post your blah.py source code?

agronholm · 2024-09-30T08:55:50Z

@samskiter ping – would you share your use case? It's hard to determine exactly what went wrong without seeing what code caused it.

davidbrochart added the bug Something isn't working label Aug 28, 2024

davidbrochart closed this as completed Aug 29, 2024

agronholm reopened this Aug 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when using Lock from thread in asyncio #773

Error when using Lock from thread in asyncio #773

davidbrochart commented Aug 28, 2024

agronholm commented Aug 29, 2024

davidbrochart commented Aug 29, 2024

smurfix commented Aug 29, 2024

davidbrochart commented Aug 29, 2024

agronholm commented Aug 29, 2024

agronholm commented Aug 29, 2024

samskiter commented Sep 26, 2024

samskiter commented Sep 26, 2024

graingert commented Sep 26, 2024 •

edited

Loading

agronholm commented Sep 30, 2024

Error when using Lock from thread in asyncio #773

Error when using Lock from thread in asyncio #773

Comments

davidbrochart commented Aug 28, 2024

Things to check first

AnyIO version

Python version

What happened?

How can we reproduce the bug?

agronholm commented Aug 29, 2024

davidbrochart commented Aug 29, 2024

smurfix commented Aug 29, 2024

davidbrochart commented Aug 29, 2024

agronholm commented Aug 29, 2024

agronholm commented Aug 29, 2024

samskiter commented Sep 26, 2024

samskiter commented Sep 26, 2024

graingert commented Sep 26, 2024 • edited Loading

agronholm commented Sep 30, 2024

graingert commented Sep 26, 2024 •

edited

Loading