[lldb] Possible design issue when using Python >= 3.12 #70453
Comments
|
@llvm/issue-subscribers-lldb Author: Tulio Magno Quites Machado Filho (tuliom)
While trying to update lldb to work with Python >= 3.12, I found the following issue:
Starting with Python 3.12, there is code now preventing the usage of This is causing the following failures: With the following output: One can reduce the list of failures to 2 by guaranteeing the critical code in |
|
@tuliom How far did you get here? I know you were able to do some work with LLDB's Python usage, is the failure list still at those 19? I assume we (the LLDB maintainers) will need to contend with this at some point in the near future, so thank you for opening this issue. |
|
FWIW I also just hit this with Python 3.12. |
@bulbazord I didn't start working on this issue. The 19 failures are remaining failures after my recent changes. I'm afraid that a larger re-design of the plugins mechanism is required in order to solve this issue. |
Gotcha. Redesigning the python plugins is no small task but it will have to be done eventually. Thanks for bringing this to our attention. |
|
On Fedora 39, with the This leaves the Scripting Bridge API essentially unusable. Of course, OP might be familiar with this, being the packager of said Fedora package... :) EDIT: One workaround is to install the |
Which sounds like a whole big mess, except for this one tantalizing note in the Python C API docs on initialization
While lldb can't control what other code has done before running python, including possibly initializing it, perhaps it's possible to reinitialize it, and therefore have access to Presumably that would only be possible when embedding Python into LLDB using the C API, where the initialization can be adjusted ahead-of-time. Whereas the extension module might be better off avoiding any attempts to meddle with the Python initialization. After all, what if the environment is an already-initialized, standalone Python interpreter that's only now loading the extension? Like in @torokati44's example of typing cc: @bulbazord |
The Python documentation [1] says that `PyImport_AppendInittab` should be called before `Py_Initialize()`. Starting with Python 3.12, this is enforced with a fatal error: Fatal Python error: PyImport_AppendInittab: PyImport_AppendInittab() may not be called after Py_Initialize() This commit ensures we only modify the table of built-in modules if Python hasn't been initialized. For Python embedded in LLDB, that means this happen exactly once, before the first call to `Py_Initialize`, which becomes a NO-OP after. However, when lldb is imported in an existing Python interpreter, Python will have already been initialized, but by definition, the lldb module will already have been loaded, so it's safe to skip adding it (again). This fixes llvm#70453. [1] https://docs.python.org/3.12/c-api/import.html#c.PyImport_AppendInittab
|
This came up again today and I decided to give this another go. I've convinced myself that it's safe (and correct) to guard the call to The two remaining unit test failures are a different issue with Python 3.12 that I haven't figured out yet. The reason I think that is because the tests don't actually use anything from the lldb module, and the issue also reproduces when skipping the call to |
The Python documentation [1] says that `PyImport_AppendInittab` should be called before `Py_Initialize()`. Starting with Python 3.12, this is enforced with a fatal error: Fatal Python error: PyImport_AppendInittab: PyImport_AppendInittab() may not be called after Py_Initialize() This commit ensures we only modify the table of built-in modules if Python hasn't been initialized. For Python embedded in LLDB, that means this happen exactly once, before the first call to `Py_Initialize`, which becomes a NO-OP after. However, when lldb is imported in an existing Python interpreter, Python will have already been initialized, but by definition, the lldb module will already have been loaded, so it's safe to skip adding it (again). This fixes llvm#70453. [1] https://docs.python.org/3.12/c-api/import.html#c.PyImport_AppendInittab
…nittests
The unit tests only test the Python objects and don't actually use
anything from the LLDB module. That means that all the additional
complexity in ScriptInterpreterPythonImpl::Initialize is overkill.
By doing the initialization by hand, we avoid the annoying
ModuleNotFoundError.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'lldb'
The error is the result of us stubbing out the SWIG (specifically
`PyInit__lldb`) because we cannot link against libLLDB from the unit
tests.
The downside of doing the initialization manually is that we do lose a
bit of test coverage. For example, issue llvm#70453 also manifested itself
in the unit tests.
|
#82098 fixes the two remaining unit tests. |
The Python documentation [1] says that `PyImport_AppendInittab` should be called before `Py_Initialize()`. Starting with Python 3.12, this is enforced with a fatal error: Fatal Python error: PyImport_AppendInittab: PyImport_AppendInittab() may not be called after Py_Initialize() This commit ensures we only modify the table of built-in modules if Python hasn't been initialized. For Python embedded in LLDB, that means this happen exactly once, before the first call to `Py_Initialize`, which becomes a NO-OP after. However, when lldb is imported in an existing Python interpreter, Python will have already been initialized, but by definition, the lldb module will already have been loaded, so it's safe to skip adding it (again). This fixes #70453. [1] https://docs.python.org/3.12/c-api/import.html#c.PyImport_AppendInittab
…nit tests (#82096) The unit tests only test the Python objects and don't actually use anything from the LLDB module. That means that all the additional complexity in ScriptInterpreterPythonImpl::Initialize is overkill. By doing the initialization by hand, we avoid the annoying ModuleNotFoundError. Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'lldb' The error is the result of us stubbing out the SWIG (specifically `PyInit__lldb`) because we cannot link against libLLDB from the unit tests. The downside of doing the initialization manually is that we do lose a bit of test coverage. For example, issue #70453 also manifested itself in the unit tests.
|
Until this is released, and gets included in the majority of the mainstream distros... Of course, one could write a small C++ program that uses |
The Python documentation [1] says that `PyImport_AppendInittab` should be called before `Py_Initialize()`. Starting with Python 3.12, this is enforced with a fatal error: Fatal Python error: PyImport_AppendInittab: PyImport_AppendInittab() may not be called after Py_Initialize() This commit ensures we only modify the table of built-in modules if Python hasn't been initialized. For Python embedded in LLDB, that means this happen exactly once, before the first call to `Py_Initialize`, which becomes a NO-OP after. However, when lldb is imported in an existing Python interpreter, Python will have already been initialized, but by definition, the lldb module will already have been loaded, so it's safe to skip adding it (again). This fixes llvm#70453. [1] https://docs.python.org/3.12/c-api/import.html#c.PyImport_AppendInittab (cherry picked from commit fbce244)
…nit tests (llvm#82096) The unit tests only test the Python objects and don't actually use anything from the LLDB module. That means that all the additional complexity in ScriptInterpreterPythonImpl::Initialize is overkill. By doing the initialization by hand, we avoid the annoying ModuleNotFoundError. Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'lldb' The error is the result of us stubbing out the SWIG (specifically `PyInit__lldb`) because we cannot link against libLLDB from the unit tests. The downside of doing the initialization manually is that we do lose a bit of test coverage. For example, issue llvm#70453 also manifested itself in the unit tests. (cherry picked from commit 5c96e71)
|
It's possible I've just got lost in a maze of twisty commits, but it looks like this fix didn't make it into LLDB 18 and also missed the cut for the LLDB 19 branch (or at least, it wasn't at the point tagged llvmorg-19-init). Please could it be merged across before LLDB 19 goes live? I've got some testing of visualisers that uses the scripting bridge to manage the tests, and I'd rather not be stuck running the tests on 17 any longer than necessary. |
The Python documentation [1] says that `PyImport_AppendInittab` should be called before `Py_Initialize()`. Starting with Python 3.12, this is enforced with a fatal error: Fatal Python error: PyImport_AppendInittab: PyImport_AppendInittab() may not be called after Py_Initialize() This commit ensures we only modify the table of built-in modules if Python hasn't been initialized. For Python embedded in LLDB, that means this happen exactly once, before the first call to `Py_Initialize`, which becomes a NO-OP after. However, when lldb is imported in an existing Python interpreter, Python will have already been initialized, but by definition, the lldb module will already have been loaded, so it's safe to skip adding it (again). This fixes llvm#70453. [1] https://docs.python.org/3.12/c-api/import.html#c.PyImport_AppendInittab (cherry picked from commit fbce244)
…nit tests (llvm#82096) The unit tests only test the Python objects and don't actually use anything from the LLDB module. That means that all the additional complexity in ScriptInterpreterPythonImpl::Initialize is overkill. By doing the initialization by hand, we avoid the annoying ModuleNotFoundError. Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'lldb' The error is the result of us stubbing out the SWIG (specifically `PyInit__lldb`) because we cannot link against libLLDB from the unit tests. The downside of doing the initialization manually is that we do lose a bit of test coverage. For example, issue llvm#70453 also manifested itself in the unit tests. (cherry picked from commit 5c96e71)
|
This also breaks python's own help function: Results in |
|
That But as far as the current LLDB Python extension goes, until this fix is released it doesn't work anyway, so it's pretty useless to have installed. The simplest solution is probably to just remove On Fedora it's |
|
@JDevlieghere Thanks for your submission. I still encounter this problem on lldb 18. It seems that the fix has not been merged into the mainline. Will it be merged in the upcoming version? cc: @bulbazord |
While trying to update lldb to work with Python >= 3.12, I found the following issue:
Starting with Python 3.12, there is code now preventing the usage of
PyImport_AppendInittabafterPy_Initialize().However, according to this source code comment:
This is causing the following failures:
With the following output:
One can reduce the list of failures to 2 by guaranteeing the critical code in
InitializePythonRAII()is executed only once.But it still doesn't solve all the cases.
The text was updated successfully, but these errors were encountered: