gh-118789: Add PyUnstable_Object_ClearWeakRefsNoCallbacks

[colesbury]

This exposes _PyWeakref_ClearWeakRefsNoCallbacks as an unstable C-API function to provide a thread-safe mechanism for clearing weakrefs without executing callbacks.

Some C-API extensions need to clear weakrefs without calling callbacks, such as after running finalizers like we do in subtype_dealloc. Previously they could use _PyWeakref_ClearRef on each weakref, but that's not thread-safe in the free-threaded build.

• Issue: _PyWeakref_ClearRef and _PyWeakref_ClearWeakRefsExceptCallbacks should be exposed in 3.13 #118789

---

📚 Documentation preview 📚: https://cpython-previews--118807.org.readthedocs.build/

[vstinner]

I suggest to rename the function to: PyObject_ClearWeakRefs().

• I don't think that it deserves to belong to the unstable API, since it's stable for many years.
• It doesn't belong to PyWeakref API, since the argument is a generic object.
• I don't think that there is an use case to call the callbacks. If tomorrow, someone comes with such use case, we can add a new function. I don't think that adding a parameter for that would be useful right now.

[vstinner]

I suggest to check if _PyType_SUPPORTS_WEAKREFS(Py_TYPE(op)) and do nothing (return) if it's not the case.

[colesbury]

I suggest to rename the function to: PyObject_ClearWeakRefs()

We already have PyObject_ClearWeakRefs: it clears the weakrefs and calls callbacks. It's the more commonly used API.

The use case for both functions is that types with user finalizers (i.e., __del__ methods) need to perform the following sequence if they want to be correct:

• Clear weakrefs and call callbacks (i.e., PyObject_ClearWeakRefs)
• Call finalizer (i.e, PyObject_CallFinalizerFromDealloc)
• Clear weakrefs again, but don't call callbacks (i.e., PyUnstable_Weakref_ClearWeakRefsExceptCallbacks)

The third step avoids leaking memory in case the finalizer added new weakrefs. We do this internally in subtype_dealloc, but extensions can't always delegate to subtype_dealloc.

Previously, the third step was implemented as:

      PyWeakReference** list =
          (PyWeakReference**)PyObject_GET_WEAKREFS_LISTPTR(self);
      while (*list)
        _PyWeakref_ClearRef(*list);
    }

But that's not thread-safe without the GIL.

It doesn't belong to PyWeakref API, since the argument is a generic object

I'll adjust the name and add the check you suggest.

[vstinner]

Why do you want to put it in the PyUnstable API?

[colesbury]

Why do you want to put it in the PyUnstable API?

That's what I understood from @Yhg1s's comments on Discord (#c-api) as the appropriate name/stability for 3.13.

I'd certainly prefer something outside of the PyUnstable API as the end state for this (i.e., in 3.14).

[vstinner]

I suggest to only do the revert in 3.13, and add the function only in 3.14.

[colesbury]

If we only add the function in 3.14, then there will be no way for C-API extensions that need this functionality to be made thread-safe in 3.13.

[colesbury]

I'll open a https://github.com/capi-workgroup/decisions issue for 3.14

[colesbury]

@encukou, would you please review this?

[vstinner]

IMO it's too late for 3.13, we're past beta1.

Suggested change

	.. versionadded:: 3.13
	.. versionadded:: 3.14

[colesbury]

@Yhg1s - is it okay to make this function public in 3.13? From our discussion on Discord in May, it sounded like you were okay with this before the RC, but I'd like to make sure.

[Yhg1s]

Yes, considering the circumstances, this is reasonable to add in beta 3.

[encukou]

Unstable API needs tests; see colesbury#2. Otherwise, this looks good to me.

@Yhg1s, do you want this in 3.13? As mentioned in the issue, projects that reimplement subtype_dealloc need this to be thread-safe in free-threading builds.
The proper solution is (IMO) to give them API so they don't need to reimplement subtype_dealloc, but that's definitely not coming in 3.13.

[vstinner]

LGTM. For the version number (.. versionadded:: in the doc), it depends if you backport the change to 3.13. If no, you should use 3.14.

[miss-islington-app]

Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

[bedevere-app]

GH-120695 is a backport of this pull request to the 3.13 branch.

[vstinner]

Congrats @colesbury :-) PyTorch should be happy with this new API if I understood correctly.