[3.12] gh-116510: Fix a crash due to shared immortal interned strings. #124541

nascheme · 2024-09-25T19:45:14Z

Fix a crash caused by immortal interned strings being shared between sub-interpreters that use basic single-phase init. In that case, the string can be used by an interpreter that outlives the interpeter that created and
interned it. For interpreters that share obmalloc state, also share the interned dict with the main interpreter.

Issue: CPython 3.12 embedded in WeeChat causes segfault on subsequent calls to Py_EndInterpreter #116510

Fix a crash caused by immortal interned strings being shared between sub-interpreters that use basic single-phase init. In that case, the string can be used by an interpreter that outlives the interpeter that created and interned it. For interpreters that share obmalloc state, also share the interned dict with the main interpreter.

nascheme · 2024-09-25T19:49:40Z

This is a small change code-wise but I'm not sure about the full implications of it. It does fix a bug but might create other bugs or problems, I'm not confident.

One downside to this change is that if many sub-interpreters are created and they create a lot of immortal interned strings, those strings will remain alive in the main interpreter state. So, it could cause a lot more memory to be used. My gut feeling is that's okay since we shouldn't be immortalizing strings unless we are fairly sure they won't cause memory bloat, even in a single interpreter case. E.g. it happens to variable names appearing in Python source code.

ericsnowcurrently

Mostly LGTM

I think there are just a couple spots where we need to clear the PyInterpreterState field for the shared interned dict case.

Objects/unicodeobject.c

bedevere-app · 2024-09-26T20:48:15Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

nascheme · 2024-09-27T00:48:53Z

I have made the requested changes; please review again

bedevere-app · 2024-09-27T00:48:58Z

Thanks for making the requested changes!

@ericsnowcurrently: please review the changes made to this pull request.

ericsnowcurrently

LGTM

bedevere-bot · 2024-09-27T07:24:10Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Arch Linux TraceRefs 3.12 has failed when building commit 5dd07eb.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1197/builds/893) and take a look at the build logs.
Check if the failure is related to this commit (5dd07eb) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1197/builds/893

Failed tests:

test_embed
test_syslog
test_threading
test_import
test_multibytecodec
test_atexit
test_importlib
test_int
test_capi

Failed subtests:

test_subinterps_different_ids - test.test_embed.EmbeddingTests.test_subinterps_different_ids
test_single_init_extension_compat - test.test_import.SubinterpImportTests.test_single_init_extension_compat
test_python_compat - test.test_import.SubinterpImportTests.test_python_compat
test_subinterps_distinct_state - test.test_embed.EmbeddingTests.test_subinterps_distinct_state
test_subinterps_main - test.test_embed.EmbeddingTests.test_subinterps_main
test_multi_init_extension_compat - test.test_import.SubinterpImportTests.test_multi_init_extension_compat

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_broken_parent_from) ... ok


TracebackTests.test_broken_submodule) ... ok


TracebackTests.test_import_bug) ... ok


TracebackTests.test_syntax_error) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 174, in test_subinterps_main
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f22e05ffe80\nobject refcount : 0\nobject type     : 0x55b39aff42c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f22e05ffe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055b39b168908)\n\nCurrent thread 0x00007f22e0989740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 180, in test_subinterps_different_ids
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f48cf32fe80\nobject refcount : 0\nobject type     : 0x557a9680d2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f48cf32fe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x0000557a96981908)\n\nCurrent thread 0x00007f48cf6b9740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_exec_failure_nested) ... ok


TracebackTests.test_nonexistent_module_nested) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1897, in test_multi_init_extension_compat
    self.check_compatible_fresh(module, strict=True)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1817, in check_compatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n    \'_testmultiphase\' in sys.builtin_module_names or\n    \'_testmultiphase\' not in sys.modules\n), repr(\'_testmultiphase\')\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import _testmultiphase\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1886, in test_single_init_extension_compat
    self.check_incompatible_fresh(module)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1841, in check_incompatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert \'_testsinglephase\' not in sys.modules, \'_testsinglephase\'\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import _testsinglephase\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 188, in test_subinterps_distinct_state
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7fef8cfbfe80\nobject refcount : 0\nobject type     : 0x55da79f132c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7fef8cfbfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055da7a087908)\n\nCurrent thread 0x00007fef8d34b740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_broken_from) ... ok


TracebackTests.test_nonexistent_module) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 174, in test_subinterps_main
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7fc0394bfe80\nobject refcount : 0\nobject type     : 0x55c76866b2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7fc0394bfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055c7687df908)\n\nCurrent thread 0x00007fc03984b740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1945, in test_python_compat
    self.check_compatible_fresh(module, strict=True)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1817, in check_compatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n    \'threading\' in sys.builtin_module_names or\n    \'threading\' not in sys.modules\n), repr(\'threading\')\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import threading\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


TracebackTests.test_unencodable_filename) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 188, in test_subinterps_distinct_state
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f14a6de80a0\nobject refcount : 0\nobject type     : 0x5567decca2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f14a6de80a0>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005567dee3e908)\n\nCurrent thread 0x00007f14a7168740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 180, in test_subinterps_different_ids
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f06ef537e80\nobject refcount : 0\nobject type     : 0x5610d3a722c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f06ef537e80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005610d3be6908)\n\nCurrent thread 0x00007f06ef8be740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_broken_parent) ... ok


TracebackTests.test_exec_failure) ... ok

…terned strings. (pythongh-124541)" This reverts commit 5dd07eb.

bedevere-app · 2024-10-01T01:54:22Z

GH-124814 is a backport of this pull request to the 3.12 branch.

… immortal interned strings. (gh-124541)" (#124814) Revert "[3.12] gh-116510: Fix a crash due to shared immortal interned strings. (gh-124541)" This reverts commit 5dd07eb.

bedevere-app bot added the awaiting core review label Sep 25, 2024

bedevere-app bot mentioned this pull request Sep 25, 2024

CPython 3.12 embedded in WeeChat causes segfault on subsequent calls to Py_EndInterpreter #116510

Open

nascheme changed the title ~~gh-116510: Fix a crash due to shared immortal interned strings.~~ [3.12] gh-116510: Fix a crash due to shared immortal interned strings. Sep 25, 2024

nascheme requested a review from ericsnowcurrently September 25, 2024 19:46

ericsnowcurrently requested changes Sep 26, 2024

View reviewed changes

Objects/unicodeobject.c Outdated Show resolved Hide resolved

Objects/unicodeobject.c Outdated Show resolved Hide resolved

bedevere-app bot removed the awaiting core review label Sep 26, 2024

bedevere-app bot added the awaiting changes label Sep 26, 2024

Small code cleanup as suggested by review.

82cfa96

bedevere-app bot added awaiting change review and removed awaiting changes labels Sep 27, 2024

bedevere-app bot requested a review from ericsnowcurrently September 27, 2024 00:48

ericsnowcurrently approved these changes Sep 27, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting change review labels Sep 27, 2024

nascheme merged commit 5dd07eb into python:3.12 Sep 27, 2024
29 of 32 checks passed

bedevere-app bot removed the awaiting merge label Sep 27, 2024

nascheme added a commit to nascheme/cpython that referenced this pull request Oct 1, 2024

Revert "[3.12] pythongh-116510: Fix a crash due to shared immortal in…

b2063bd

…terned strings. (pythongh-124541)" This reverts commit 5dd07eb.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[3.12] gh-116510: Fix a crash due to shared immortal interned strings. #124541

[3.12] gh-116510: Fix a crash due to shared immortal interned strings. #124541

nascheme commented Sep 25, 2024 •

edited by bedevere-app bot

Loading

nascheme commented Sep 25, 2024

ericsnowcurrently left a comment

bedevere-app bot commented Sep 26, 2024

nascheme commented Sep 27, 2024

bedevere-app bot commented Sep 27, 2024

ericsnowcurrently left a comment

bedevere-bot commented Sep 27, 2024

bedevere-app bot commented Oct 1, 2024

[3.12] gh-116510: Fix a crash due to shared immortal interned strings. #124541

[3.12] gh-116510: Fix a crash due to shared immortal interned strings. #124541

Conversation

nascheme commented Sep 25, 2024 • edited by bedevere-app bot Loading

nascheme commented Sep 25, 2024

ericsnowcurrently left a comment

Choose a reason for hiding this comment

bedevere-app bot commented Sep 26, 2024

nascheme commented Sep 27, 2024

bedevere-app bot commented Sep 27, 2024

ericsnowcurrently left a comment

Choose a reason for hiding this comment

bedevere-bot commented Sep 27, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-app bot commented Oct 1, 2024

nascheme commented Sep 25, 2024 •

edited by bedevere-app bot

Loading