RuntimeError on race on weakset concurrent iteration

[bensimner]

BPO	45278
Nosy	@freddrake, @pitrou, @serhiy-storchaka, @graingert, @bensimner
Files	weakset_concurrent_iter_runtimeerror.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-09-24.11:03:08.880>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'RuntimeError on race on weakset concurrent iteration'
updated_at = <Date 2021-09-24.11:06:46.354>
user = 'https://github.com/bensimner'

bugs.python.org fields:

activity = <Date 2021-09-24.11:06:46.354>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-09-24.11:03:08.880>
creator = 'bjs'
dependencies = []
files = ['50302']
hgrepos = []
issue_num = 45278
keywords = []
message_count = 1.0
messages = ['402553']
nosy_count = 5.0
nosy_names = ['fdrake', 'pitrou', 'serhiy.storchaka', 'graingert', 'bjs']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45278'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[bensimner]

This is a very subtle race

WeakSet uses _weakrefset.py's _IterationGuard structure to protect against the case where the elements the WeakSet refers to get cleaned up while a thread is iterating over the WeakSet.

It defers the actual removal of any elements which get gc'd during iteration until the end of the iteration.

the WeakSet keeps track of all the iterators, and waits until there are no more threads iterating over the set before it does the removal: https://github.com/python/cpython/blob/main/Lib/_weakrefset.py#L30

However there is a race, if another thread begins iterating after the `if s:` check but before the _commit_removals call has ended, that iteration can get a RuntimeError.

attached is an example script that can generate such RuntimeError's,
although the race window here is very small and so to observe yourself you may have to tweak the magic constants around.

As far as I'm aware nobody has reported seeing this bug happen in production, but some libraries (e.g. asyncio) do currently rely on concurrently iterating a weakset, so it's not implausible.

[duaneg]

This is a dup of #123089 (or vice versa). The fix for that has been merged for 3.14.0, and I believe this issue can also be closed.