Race condition in WeakKeyDictionary/WeakValueDictionary

[colesbury]

BPO	45809
Nosy	@colesbury
Files	issue45809-repro.patch: Patch to make reproduction easier

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-15.19:02:38.373>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'Race condition in WeakKeyDictionary/WeakKeyDictionary'
updated_at = <Date 2021-11-15.19:05:22.656>
user = 'https://github.com/colesbury'

bugs.python.org fields:

activity = <Date 2021-11-15.19:05:22.656>
actor = 'colesbury'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-11-15.19:02:38.373>
creator = 'colesbury'
dependencies = []
files = ['50440']
hgrepos = []
issue_num = 45809
keywords = ['patch']
message_count = 2.0
messages = ['406357', '406358']
nosy_count = 1.0
nosy_names = ['colesbury']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45809'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-89967: make WeakKeyDictionary and WeakValueDictionary thread safe #125325

[colesbury]

The issue described bpo-7105 (and maybe bpo-7060) still exists due to a race condition in WeakKeyDictionary. This shows up as test failure that looks like:

  test test_weakref failed -- Traceback (most recent call last):
    File "Lib/test/test_weakref.py", line 1960, in test_threaded_weak_value_dict_deepcopy
      self.check_threaded_weak_dict_copy(weakref.WeakValueDictionary, True)
    File "Lib/test/test_weakref.py", line 1940, in check_threaded_weak_dict_copy
      raise exc[0]
    File "Lib/test/test_weakref.py", line 1897, in dict_copy
      _ = copy.deepcopy(d)
    File "Lib/copy.py", line 153, in deepcopy
      y = copier(memo)
    File "Lib/weakref.py", line 189, in __deepcopy__
      for key, wr in self.data.items():
  RuntimeError: dictionary changed size during iteration

The cause is that the check of "self._iterating" and the call to "_atomic_removal" are not performed atomically together. By the time _atomic_removal() is called, an iteration might have already started.

cpython/Lib/weakref.py

Lines 109 to 114 in ec382fa

	if self._iterating:
	self._pending_removals.append(wr.key)
	else:
	# Atomic removal is necessary since this function
	# can be called asynchronously by the GC
	_atomic_removal(self.data, wr.key)

[colesbury]

The attached patch (bpo-45809-repro.patch) introduces artificial delays to make reproduction of the underlying issue easier.

To reproduce the issue:

patch -p1 < bpo-45809-repro.patch
./python -m test test_weakref -m test_threaded_weak_value_dict_deepcopy -v