Race in concurrent list mutation and item retrieval

[Yhg1s]

Bug report

Bug description:

A data race exists in lists when they are concurrently updated and indexed (or otherwise accessed): for slice assignment and other ways of adding or removing multiple items (list.extend, in-place multiplication), memory writes to the newly allocated areas happen with memcpy/memmove, which have no write size/atomicity guarantees (and it's not unlikely that they will perform jagged writes). These reallocations happen with the lock held, but since we're relying on QSBR, concurrent reads of those memory areas are technically possible.

The tests in PR #128798 (test_free_threading.test_iteration), when run under ThreadSanitizer, expose those races. (They are suppressed in Tools/tsan/suppressions_free_threading.txt).

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-129069: make list ass_slice and memory_repeat safe in free-threading #131882

[colesbury]

I'm a bit confused by the "memory writes to the newly allocated areas happen with memcpy/memmove" part of the issue description.

The data races with memcpy I've seen involve writes to the existing item array. For example, in list_ass_slice_lock_held, we have memmove calls with self->ob_item and recycle as the destinations. The ones with the newly allocated recycle as the destination are okay, but the ones with self->ob_item are potentially a problem.

[Yhg1s]

Yes, you're right, 'newly allocated' isn't really relevant. I just meant that the calls that move items around before/after list_resize() are potentially problematic. I have to admit my head hurts every time I try to trace through list_ass_slice_lock_held.