Add per-interpreter storage for gil_safe_call_once_and_store

[XuehaiPan]

Description

Fixes #5926

For !defined(PYBIND11_HAS_SUBINTERPRETER_SUPPORT), the previous behavior is unchanged.

For defined(PYBIND11_HAS_SUBINTERPRETER_SUPPORT), the pybind11 internals is already a per-interpreter-dependent singleton stored in the interpreter's state dict. This PR adds a new map to the interpreter-dependent internals from the gil_safe_call_once_and_store's address to the result, where the gil_safe_call_once_and_store is a static object that has a fixed address.

UPDATE: The storage map is moved to a separate capsule in the per-interpreter state dict instead of putting it as a member of internals. Because the per-interpreter internals are leaked in the program, the storage map is never GC-ed on (sub-)interpreter shutdown.

Suggested changelog entry:

• Add per-interpreter storage for gil_safe_call_once_and_store to make it safe under multi-interpreters.

[XuehaiPan]

I have updated the approach to move the storage map as a separate capsule in the per-interpreter state dict instead of putting it in the internals. The PR description is also updated.

The CI tests are looking good so far. But I found some concurrency bugs for this patch in my downstream PR metaopt/optree#245. I'm not sure if it is coming from pybind11 or the Python core itself.

• feat: add subinterpreters support metaopt/optree#245

I will do further investigation about these.

---

Bug 1: Duplicate native enum registration.

ImportError: pybind11::native_enum<...>("PyTreeKind") is already registered!

Test Case

def test_import_in_subinterpreter_before_main():
    # OK
    check_script_in_subprocess(
        textwrap.dedent(
            """
            import contextlib
            import gc
            from concurrent import interpreters

            subinterpreter = None
            with contextlib.closing(interpreters.create()) as subinterpreter:
                subinterpreter.exec('import optree')

            import optree

            del optree, subinterpreter
            for _ in range(10):
                gc.collect()
            """,
        ).strip(),
        output='',
        rerun=NUM_FLAKY_RERUNS,
    )

    # FAIL
    check_script_in_subprocess(
        textwrap.dedent(
            f"""
            import contextlib
            import gc
            import random
            from concurrent import interpreters

            subinterpreter = subinterpreters = stack = None
            with contextlib.ExitStack() as stack:
                subinterpreters = [
                    stack.enter_context(contextlib.closing(interpreters.create()))
                    for _ in range({NUM_FUTURES})
                ]
                random.shuffle(subinterpreters)
                for subinterpreter in subinterpreters:
                    subinterpreter.exec('import optree')

            import optree

            del optree, subinterpreter, subinterpreters, stack
            for _ in range(10):
                gc.collect()
            """,
        ).strip(),
        output='',
        rerun=NUM_FLAKY_RERUNS,
    )

    # FAIL
    check_script_in_subprocess(
        textwrap.dedent(
            f"""
            import contextlib
            import gc
            import random
            from concurrent import interpreters

            subinterpreter = subinterpreters = stack = None
            with contextlib.ExitStack() as stack:
                subinterpreters = [
                    stack.enter_context(contextlib.closing(interpreters.create()))
                    for _ in range({NUM_FUTURES})
                ]
                random.shuffle(subinterpreters)
                for subinterpreter in subinterpreters:
                    subinterpreter.exec('import optree')

                import optree

            del optree, subinterpreter, subinterpreters, stack
            for _ in range(10):
                gc.collect()
            """,
        ).strip(),
        output='',
        rerun=NUM_FLAKY_RERUNS,
    )

Traceback

__________________________________________________________________ test_import_in_subinterpreter_before_main ___________________________________________________________________

    def test_import_in_subinterpreter_before_main():
        check_script_in_subprocess(
            textwrap.dedent(
                """
                import contextlib
                import gc
                from concurrent import interpreters
    
                subinterpreter = None
                with contextlib.closing(interpreters.create()) as subinterpreter:
                    subinterpreter.exec('import optree')
    
                import optree
    
                del optree, subinterpreter
                for _ in range(10):
                    gc.collect()
                """,
            ).strip(),
            output='',
            rerun=NUM_FLAKY_RERUNS,
        )
    
>       check_script_in_subprocess(
            textwrap.dedent(
                f"""
                import contextlib
                import gc
                import random
                from concurrent import interpreters
    
                subinterpreter = subinterpreters = stack = None
                with contextlib.ExitStack() as stack:
                    subinterpreters = [
                        stack.enter_context(contextlib.closing(interpreters.create()))
                        for _ in range({NUM_FUTURES})
                    ]
                    random.shuffle(subinterpreters)
                    for subinterpreter in subinterpreters:
                        subinterpreter.exec('import optree')
    
                import optree
    
                del optree, subinterpreter, subinterpreters, stack
                for _ in range(10):
                    gc.collect()
                """,
            ).strip(),
            output='',
            rerun=NUM_FLAKY_RERUNS,
        )


concurrent/test_subinterpreters.py:267: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

script = "import contextlib\nimport gc\nimport random\nfrom concurrent import interpreters\n\nsubinterpreter = subinterpreters ...optree')\n\nimport optree\n\ndel optree, subinterpreter, subinterpreters, stack\nfor _ in range(10):\n    gc.collect()"

    def check_script_in_subprocess(script, /, *, output, env=None, cwd=TEST_ROOT, rerun=1):
        result = ''
        for _ in range(rerun):
            try:
                result = subprocess.check_output(
                    [sys.executable, '-Walways', '-Werror', '-c', script],
                    stderr=subprocess.STDOUT,
                    text=True,
                    encoding='utf-8',
                    cwd=cwd,
                    env={
                        key: value
                        for key, value in (env if env is not None else os.environ).items()
                        if not key.startswith(('PYTHON', 'PYTEST', 'COV_'))
                    },
                )
            except subprocess.CalledProcessError as ex:
>               raise CalledProcessError(ex.returncode, ex.cmd, ex.output, ex.stderr) from None
E               helpers.CalledProcessError: Command '['/home/PanXuehai/Projects/optree/venv/bin/python3', '-Walways', '-Werror', '-c', "import contextlib\nimport gc\nimport random\nfrom concurrent import interpreters\n\nsubinterpreter = subinterpreters = stack = None\nwith contextlib.ExitStack() as stack:\n    subinterpreters = [\n        stack.enter_context(contextlib.closing(interpreters.create()))\n        for _ in range(16)\n    ]\n    random.shuffle(subinterpreters)\n    for subinterpreter in subinterpreters:\n        subinterpreter.exec('import optree')\n\nimport optree\n\ndel optree, subinterpreter, subinterpreters, stack\nfor _ in range(10):\n    gc.collect()"]' returned non-zero exit status 1.
E               Output:
E               Traceback (most recent call last):
E                 File "<string>", line 16, in <module>
E                   import optree
E                 File "/home/PanXuehai/Projects/optree/optree/__init__.py", line 17, in <module>
E                   from optree import accessors, dataclasses, functools, integrations, pytree, treespec, typing
E                 File "/home/PanXuehai/Projects/optree/optree/accessors.py", line 25, in <module>
E                   import optree._C as _C
E               ImportError: pybind11::native_enum<...>("PyTreeKind") is already registered!

_          = 0
cwd        = PosixPath('/home/PanXuehai/Projects/optree/tests')
env        = None
output     = ''
rerun      = 8
result     = ''
script     = "import contextlib\nimport gc\nimport random\nfrom concurrent import interpreters\n\nsubinterpreter = subinterpreters ...optree')\n\nimport optree\n\ndel optree, subinterpreter, subinterpreters, stack\nfor _ in range(10):\n    gc.collect()"

helpers.py:187: CalledProcessError

---

Bug 2: Thread-safety issues for multiple-interpreters are ever created.

• Case 1: No multiple interpreters, only the main interpreter. Everything works fine.

  # Run tests separately
  make test PYTESTOPTS='-k "subinterpreter"'       # OK, no threading concurrency tests
  make test PYTESTOPTS='-k "not subinterpreter"'   # OK, the program only has the main interpreter

• Case 2: Create multiple interpreters, and destroy them before doing threading concurrency tests (only the main interpreter is alive during the concurrency tests).

  The C++ objects are locked with critical sections, and the GIL is enabled during the test.

  # There is only the main interpreter alive during threading concurrency tests
  #
  #     ```python
  #     with contextlib.closing(concurrent.interpreters.create()) as subintrepreter:
  #         # do something
  #         ...
  #
  #     # at this point, only the main interpreter is alive
  #
  #     # threading tests
  #     ...
  #     ```
  
  # Run tests alltogather
  make test  # subinterpreter tests all passed, but threading concurrency tests produce incorrect results