methodcaller is not thread-safe (or re-entrant)

[colesbury]

Bug report

EDIT: edited to clarify that the issue is in the C implementation of operator.methodcaller.

Originally reported by @ngoldbaum in crate-py/rpds#101

Reproducer

from operator import methodcaller

from concurrent.futures import ThreadPoolExecutor

class HashTrieMap():
    def keys(self):
        return None
    def values(self):
        return None
    def items(self):
        return None

num_workers=1000

views = [methodcaller(p) for p in ["keys", "values", "items"]]

def work(view):
    m, d = HashTrieMap(), {}
    view(m)
    view(d)

iterations = 10

for _ in range(iterations):
    
    executor = ThreadPoolExecutor(max_workers=num_workers)

    for view in views:
        futures = [executor.submit(work, view) for _ in range(num_workers)]
        results = [future.result() for future in futures]

Once every 5-10 runs, the program prints:

TypeError: descriptor 'keys' for 'dict' objects doesn't apply to a 'HashTrieMap' object

The problem is that operator.methodcaller is not thread-safe because it modifies the vectorcall_args, which is shared across calls:

cpython/Modules/_operator.c

Lines 1646 to 1666 in 0af4ec3

	static PyObject *
	methodcaller_vectorcall(
	methodcallerobject *mc, PyObject *const *args, size_t nargsf, PyObject* kwnames)
	{
	if (!_PyArg_CheckPositional("methodcaller", PyVectorcall_NARGS(nargsf), 1, 1)
	|| !_PyArg_NoKwnames("methodcaller", kwnames)) {
	return NULL;
	}
	if (mc->vectorcall_args == NULL) {
	if (_methodcaller_initialize_vectorcall(mc) < 0) {
	return NULL;
	}
	}
	assert(mc->vectorcall_args != 0);
	mc->vectorcall_args[0] = args[0];
	return PyObject_VectorcallMethod(
	mc->name, mc->vectorcall_args,
	(PyTuple_GET_SIZE(mc->xargs)) | PY_VECTORCALL_ARGUMENTS_OFFSET,
	mc->vectorcall_kwnames);
	}

I think this is generally unsafe, not just for free threading. The vectorcall args array needs to be valid for the duration of the call, and it's possible for methodcaller to be called reentrantly or by another thread while the call is still ongoing.

Linked PRs

• gh-127065: Make methodcaller thread-safe in free threading build #127109

[colesbury]

Hmm, actually it looks like a thread safety issue in methodcaller.

[ngoldbaum]

Yeah, I noticed early it wasn't triggering if I called the descriptors directly without going through methodcaller.

[colesbury]

I think the optimized vectorcall implementation from #107201 is not reentrant or thread-safe (even with the GIL) because the shared vectorcall_args array is modified during each invocation:

cpython/Modules/_operator.c

Line 1661 in 3926842

mc->vectorcall_args[0] = args[0];

Most (or all) of our current vectorcall implementations extract out args[0] early, but that's not guaranteed and the vectorcall protocol is a public API.

For now, I'm planning to just disable the optimization in the free threading build (#127109), but I don't see an easy way of making it thread-safe, and I think we should consider reverting #107201 in 3.13 and main.

cc @eendebakpt @corona10 @vstinner