[Feature Request] Subinterpreters Support (PEP 554 & PEP 684)

[XuehaiPan]

Required prerequisites

• I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)

Motivation

The integration of subinterpreters support in optree, aligned with PEP 554 – Multiple Interpreters in the Stdlib and PEP 684 – A Per-Interpreter GIL, is motivated by the evolving landscape of Python's concurrency and multi-threading capabilities. Python 3.12's introduction of a per-interpreter GIL and the anticipated standard library support for multiple interpreters in Python 3.13 signify a shift towards more sophisticated concurrency models. This shift presents an opportunity for optree to stay at the forefront of Python's high-performance computing.

Incorporating subinterpreters support is crucial for optree to harness the benefits of improved isolation and concurrency control in Python. This enhancement will enable optree to better serve applications that require robust multi-threaded processing, particularly in high-performance computing environments.

The primary challenges in this endeavor include:

Managing the PyTree Registry in a Multi-Interpreter Context: Ensuring each subinterpreter has its own isolated PyTree registry is critical for thread safety and data integrity. Investigating efficient ways to manage these registries – their creation, access, and destruction – in tandem with the lifecycle of each subinterpreter, is a key challenge.

Ensuring Compatibility and Performance: The implementation must not only be compatible with the new interpreter architecture but also maintain, if not enhance, the current performance benchmarks of optree. Balancing compatibility with performance in a multi-interpreter setup will be a significant area of investigation.

Isolation vs. Interoperability: While isolation is a core benefit of subinterpreters, ensuring that optree remains interoperable and efficient across different interpreter instances is essential. This involves exploring mechanisms for data sharing and communication between PyTrees across subinterpreters without compromising on the isolation benefits.

By addressing these challenges, optree aims to leverage Python's advancements in concurrency, ensuring that the library remains a top choice for developers working in multi-threaded and high-performance computing scenarios.

Solution

No response

Alternatives

No response

Additional context

References:

• PEP 684 – A Per-Interpreter GIL
• PEP 554 – Multiple Interpreters in the Stdlib
• gh-99113: A Per-Interpreter GIL! python/cpython#104210
• [feature request] PyTorch support for sub-interpreters with PEP 684 accepted and release in Python 3.12 pytorch/pytorch#102517
• [pytree] test C++ pytree compatibility with multipy pytorch/pytorch#112477
• https://github.com/ericsnowcurrently/multi-core-python/wiki/0-The-Plan