Declare free-threaded support for PyModule (#4588)

* WIP: declare free-threaded support in pymodule macro

* ignore ruff lint about unused import

* eliminate gil re-enabling in pytests

* fix clippy nit

* fix return type of PyUnstable_Module_SetGIL binding

* add a way to declare free-threaded support without macros

* fix ruff

* fix changed ui test answer

* fix build issues on old python versions

* fix runtime warnings in examples

* ensure that the GIL does not get re-enabled in the pytests

* add changelog entry

* fix ruff

* fix compiler error on older pythons

* fix clippy

* really fix clippy and expose supports_free_threaded on all builds

* fix clippy and msrv

* fix examples on gil-disabled python

* fix free-threaded clippy

* fix unused import in example

* Add pyo3-build-config as a build dependency to examples that need it

* add docs

* add rust tests so coverage picks up the new code

* fix some formatting issues

* Apply cleanups

* fix cargo fmt --check

* revert changes to non-FFI examples

* apply David's suggestion for the guide

* link to raw FFI examples in the guide

* fix config guards in moduleobject.rs

* rename supports_free_threaded to gil_used

* remove ensure_gil_enabled from pyo3-ffi/build.rs

* update docs for PyModule::gil_used

* remove UNSAFE_PYO3_BUILD_FREE_THREADED from the CI config

* fix merge conflict screwup

* fix nox -s test-py

* fix guide links

* remove redundant pytest test

* fix issue with wrap_pymodule not respecting user choice for GIL support

* replace map.unwrap_or with map_or

* fix refcounting error in ffi example

.github/workflows/ci.yml

@@ -553,8 +553,6 @@ jobs:
   test-free-threaded:
     needs: [fmt]
     runs-on: ubuntu-latest
-    env:
-      UNSAFE_PYO3_BUILD_FREE_THREADED: 1
     steps:
       - uses: actions/checkout@v4
       - uses: Swatinem/rust-cache@v2

guide/src/free-threading.md

@@ -38,11 +38,83 @@ native Python runtime by building on the Rust [`Send` and
 `Sync`](https://doc.rust-lang.org/nomicon/send-and-sync.html) traits.
 
 This document provides advice for porting Rust code using PyO3 to run under
-free-threaded Python. While many simple PyO3 uses, like defining an immutable
-Python class, will likely work "out of the box", there are currently some
-limitations.
+free-threaded Python.
+
+## Supporting free-threaded Python with PyO3
+
+Many simple uses of PyO3, like exposing bindings for a "pure" Rust function
+with no side-effects or defining an immutable Python class, will likely work
+"out of the box" on the free-threaded build. All that will be necessary is to
+annotate Python modules declared by rust code in your project to declare that
+they support free-threaded Python, for example by declaring the module with
+`#[pymodule(gil_used = false)]`.
+
+At a low-level, annotating a module sets the `Py_MOD_GIL` slot on modules
+defined by an extension to `Py_MOD_GIL_NOT_USED`, which allows the interpreter
+to see at runtime that the author of the extension thinks the extension is
+thread-safe. You should only do this if you know that your extension is
+thread-safe. Because of Rust's guarantees, this is already true for many
+extensions, however see below for more discussion about how to evaluate the
+thread safety of existing Rust extensions and how to think about the PyO3 API
+using a Python runtime with no GIL.
+
+If you do not explicitly mark that modules are thread-safe, the Python
+interpreter will re-enable the GIL at runtime and print a `RuntimeWarning`
+explaining which module caused it to re-enable the GIL. You can also force the
+GIL to remain disabled by setting the `PYTHON_GIL=0` as an environment variable
+or passing `-Xgil=0` when starting Python (`0` means the GIL is turned off).
+
+If you are sure that all data structures exposed in a `PyModule` are
+thread-safe, then pass `gil_used = false` as a parameter to the
+`pymodule` procedural macro declaring the module or call
+`PyModule::gil_used` on a `PyModule` instance.  For example:
 
-## Many symbols exposed by PyO3 have `GIL` in the name
+```rust
+use pyo3::prelude::*;
+
+/// This module supports free-threaded Python
+#[pymodule(gil_used = false)]
+fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> {
+    // add members to the module that you know are thread-safe
+    Ok(())
+}
+```
+
+Or for a module that is set up without using the `pymodule` macro:
+
+```rust
+use pyo3::prelude::*;
+
+# #[allow(dead_code)]
+fn register_child_module(parent_module: &Bound<'_, PyModule>) -> PyResult<()> {
+    let child_module = PyModule::new(parent_module.py(), "child_module")?;
+    child_module.gil_used(false)?;
+    parent_module.add_submodule(&child_module)
+}
+
+```
+
+See the
+[`string-sum`](https://github.com/PyO3/pyo3/tree/main/pyo3-ffi/examples/string-sum)
+example for how to declare free-threaded support using raw FFI calls for modules
+using single-phase initialization and the
+[`sequential`](https://github.com/PyO3/pyo3/tree/main/pyo3-ffi/examples/sequential)
+example for modules using multi-phase initialization.
+
+## Special considerations for the free-threaded build
+
+The free-threaded interpreter does not have a GIL, and this can make interacting
+with the PyO3 API confusing, since the API was originally designed around strong
+assumptions about the GIL providing locking.  Additionally, since the GIL
+provided locking for operations on Python objects, many existing extensions that
+provide mutable data structures relied on the GIL to make interior mutability
+thread-safe.
+
+Working with PyO3 under the free-threaded interpreter therefore requires some
+additional care and mental overhead compared with a GIL-enabled interpreter. We
+discuss how to handle this below.
+
+### Many symbols exposed by PyO3 have `GIL` in the name
 
 We are aware that there are some naming issues in the PyO3 API that make it
 awkward to think about a runtime environment where there is no GIL. We plan to
@@ -87,7 +159,7 @@ garbage collector can only run if all threads are detached from the runtime (in
 a stop-the-world state), so detaching from the runtime allows freeing unused
 memory.
 
-## Exceptions and panics for multithreaded access of mutable `pyclass` instances
+### Exceptions and panics for multithreaded access of mutable `pyclass` instances
 
 Data attached to `pyclass` instances is protected from concurrent access by a
 `RefCell`-like pattern of runtime borrow checking. Like a `RefCell`, PyO3 will
@@ -104,7 +176,7 @@ The most straightforward way to trigger this problem to use the Python
 [`pyclass`]({{#PYO3_DOCS_URL}}/pyo3/attr.pyclass.html). For example,
 consider the following implementation:
 
-```
+```rust
 # use pyo3::prelude::*;
 # fn main() {
 #[pyclass]
@@ -211,7 +283,7 @@ Python::with_gil(|py| {
 # }
 ```
 
-## `GILProtected` is not exposed
+### `GILProtected` is not exposed
 
 [`GILProtected`] is a PyO3 type that allows mutable access to static data by
 leveraging the GIL to lock concurrent access from other threads. In

newsfragments/4588.added.md

@@ -0,0 +1,3 @@
+* It is now possible to declare that a module supports the free-threaded build
+  by either calling `PyModule::gil_used` or passing
+  `gil_used = false` as a parameter to the `pymodule` proc macro.

noxfile.py

@@ -676,14 +676,6 @@ def test_version_limits(session: nox.Session):
         config_file.set("PyPy", "3.11")
         _run_cargo(session, "check", env=env, expect_error=True)
 
-        # Python build with GIL disabled should fail building
-        config_file.set("CPython", "3.13", build_flags=["Py_GIL_DISABLED"])
-        _run_cargo(session, "check", env=env, expect_error=True)
-
-        # Python build with GIL disabled should pass with env flag on
-        env["UNSAFE_PYO3_BUILD_FREE_THREADED"] = "1"
-        _run_cargo(session, "check", env=env)
-
 
 @nox.session(name="check-feature-powerset", venv_backend="none")
 def check_feature_powerset(session: nox.Session):

pyo3-ffi/build.rs

@@ -6,7 +6,6 @@ use pyo3_build_config::{
     },
     warn, BuildFlag, PythonImplementation,
 };
-use std::ops::Not;
 
 /// Minimum Python version PyO3 supports.
 struct SupportedVersions {
@@ -107,7 +106,17 @@ fn ensure_python_version(interpreter_config: &InterpreterConfig) -> Result<()> {
 
     if interpreter_config.abi3 {
         match interpreter_config.implementation {
-            PythonImplementation::CPython => {}
+            PythonImplementation::CPython => {
+                if interpreter_config
+                    .build_flags
+                    .0
+                    .contains(&BuildFlag::Py_GIL_DISABLED)
+                {
+                    warn!(
+                            "The free-threaded build of CPython does not yet support abi3 so the build artifacts will be version-specific."
+                        )
+                }
+            }
             PythonImplementation::PyPy => warn!(
                 "PyPy does not yet support abi3 so the build artifacts will be version-specific. \
                 See https://github.com/pypy/pypy/issues/3397 for more information."
@@ -121,29 +130,6 @@ fn ensure_python_version(interpreter_config: &InterpreterConfig) -> Result<()> {
     Ok(())
 }
 
-fn ensure_gil_enabled(interpreter_config: &InterpreterConfig) -> Result<()> {
-    let gil_enabled = interpreter_config
-        .build_flags
-        .0
-        .contains(&BuildFlag::Py_GIL_DISABLED)
-        .not();
-    ensure!(
-        gil_enabled || std::env::var("UNSAFE_PYO3_BUILD_FREE_THREADED").map_or(false, |os_str| os_str == "1"),
-        "the Python interpreter was built with the GIL disabled, which is not yet supported by PyO3\n\
-        = help: see https://github.com/PyO3/pyo3/issues/4265 for more information\n\
-        = help: please check if an updated version of PyO3 is available. Current version: {}\n\
-        = help: set UNSAFE_PYO3_BUILD_FREE_THREADED=1 to suppress this check and build anyway for free-threaded Python",
-        std::env::var("CARGO_PKG_VERSION").unwrap()
-    );
-    if !gil_enabled && interpreter_config.abi3 {
-        warn!(
-            "The free-threaded build of CPython does not yet support abi3 so the build artifacts will be version-specific."
-        )
-    }
-
-    Ok(())
-}
-
 fn ensure_target_pointer_width(interpreter_config: &InterpreterConfig) -> Result<()> {
     if let Some(pointer_width) = interpreter_config.pointer_width {
         // Try to check whether the target architecture matches the python library
@@ -209,7 +195,6 @@ fn configure_pyo3() -> Result<()> {
 
     ensure_python_version(&interpreter_config)?;
     ensure_target_pointer_width(&interpreter_config)?;
-    ensure_gil_enabled(&interpreter_config)?;
 
     // Serialize the whole interpreter config into DEP_PYTHON_PYO3_CONFIG env var.
     interpreter_config.to_cargo_dep_env()?;

pyo3-ffi/examples/sequential/Cargo.toml

@@ -10,4 +10,7 @@ crate-type = ["cdylib", "lib"]
 [dependencies]
 pyo3-ffi = { path = "../../", features = ["extension-module"] }
 
+[build-dependencies]
+pyo3-build-config = { path = "../../../pyo3-build-config" }
+
 [workspace]

pyo3-ffi/examples/sequential/build.rs

@@ -0,0 +1,3 @@
+fn main() {
+    pyo3_build_config::use_pyo3_cfgs();
+}

pyo3-ffi/examples/sequential/src/module.rs

@@ -23,6 +23,11 @@ static mut SEQUENTIAL_SLOTS: &[PyModuleDef_Slot] = &[
         slot: Py_mod_multiple_interpreters,
         value: Py_MOD_PER_INTERPRETER_GIL_SUPPORTED,
     },
+    #[cfg(Py_GIL_DISABLED)]
+    PyModuleDef_Slot {
+        slot: Py_mod_gil,
+        value: Py_MOD_GIL_NOT_USED,
+    },
     PyModuleDef_Slot {
         slot: 0,
         value: ptr::null_mut(),

pyo3-ffi/examples/string-sum/Cargo.toml

@@ -10,4 +10,7 @@ crate-type = ["cdylib"]
 [dependencies]
 pyo3-ffi = { path = "../../", features = ["extension-module"] }
 
+[build-dependencies]
+pyo3-build-config = { path = "../../../pyo3-build-config" }
+
 [workspace]

pyo3-ffi/examples/string-sum/build.rs

@@ -0,0 +1,3 @@
+fn main() {
+    pyo3_build_config::use_pyo3_cfgs();
+}

pyo3-ffi/examples/string-sum/src/lib.rs

@@ -32,7 +32,18 @@ static mut METHODS: &[PyMethodDef] = &[
 #[allow(non_snake_case)]
 #[no_mangle]
 pub unsafe extern "C" fn PyInit_string_sum() -> *mut PyObject {
-    PyModule_Create(ptr::addr_of_mut!(MODULE_DEF))
+    let module = PyModule_Create(ptr::addr_of_mut!(MODULE_DEF));
+    if module.is_null() {
+        return module;
+    }
+    #[cfg(Py_GIL_DISABLED)]
+    {
+        if PyUnstable_Module_SetGIL(module, Py_MOD_GIL_NOT_USED) < 0 {
+            Py_DECREF(module);
+            return std::ptr::null_mut();
+        }
+    }
+    module
 }
 
 /// A helper to parse function arguments

pyo3-ffi/src/moduleobject.rs

@@ -88,6 +88,10 @@ pub const Py_mod_create: c_int = 1;
 pub const Py_mod_exec: c_int = 2;
 #[cfg(Py_3_12)]
 pub const Py_mod_multiple_interpreters: c_int = 3;
+#[cfg(Py_3_13)]
+pub const Py_mod_gil: c_int = 4;
+
+// skipped private _Py_mod_LAST_SLOT
 
 #[cfg(Py_3_12)]
 pub const Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED: *mut c_void = 0 as *mut c_void;
@@ -96,7 +100,15 @@ pub const Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED: *mut c_void = 1 as *mut c_void
 #[cfg(Py_3_12)]
 pub const Py_MOD_PER_INTERPRETER_GIL_SUPPORTED: *mut c_void = 2 as *mut c_void;
 
-// skipped non-limited _Py_mod_LAST_SLOT
+#[cfg(Py_3_13)]
+pub const Py_MOD_GIL_USED: *mut c_void = 0 as *mut c_void;
+#[cfg(Py_3_13)]
+pub const Py_MOD_GIL_NOT_USED: *mut c_void = 1 as *mut c_void;
+
+#[cfg(all(not(Py_LIMITED_API), Py_GIL_DISABLED))]
+extern "C" {
+    pub fn PyUnstable_Module_SetGIL(module: *mut PyObject, gil: *mut c_void) -> c_int;
+}
 
 #[repr(C)]
 pub struct PyModuleDef {

pyo3-macros-backend/src/attributes.rs

@@ -7,7 +7,7 @@ use syn::{
     punctuated::Punctuated,
     spanned::Spanned,
     token::Comma,
-    Attribute, Expr, ExprPath, Ident, Index, LitStr, Member, Path, Result, Token,
+    Attribute, Expr, ExprPath, Ident, Index, LitBool, LitStr, Member, Path, Result, Token,
 };
 
 pub mod kw {
@@ -44,6 +44,7 @@ pub mod kw {
     syn::custom_keyword!(transparent);
     syn::custom_keyword!(unsendable);
     syn::custom_keyword!(weakref);
+    syn::custom_keyword!(gil_used);
 }
 
 fn take_int(read: &mut &str, tracker: &mut usize) -> String {
@@ -308,6 +309,7 @@ pub type RenameAllAttribute = KeywordAttribute<kw::rename_all, RenamingRuleLitSt
 pub type StrFormatterAttribute = OptionalKeywordAttribute<kw::str, StringFormatter>;
 pub type TextSignatureAttribute = KeywordAttribute<kw::text_signature, TextSignatureAttributeValue>;
 pub type SubmoduleAttribute = kw::submodule;
+pub type GILUsedAttribute = KeywordAttribute<kw::gil_used, LitBool>;
 
 impl<K: Parse + std::fmt::Debug, V: Parse> Parse for KeywordAttribute<K, V> {
     fn parse(input: ParseStream<'_>) -> Result<Self> {