Skip to content

Push down the swig module to avoid an import cycle #129135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 3, 2025

Conversation

dingxiangfei2009
Copy link
Contributor

Fix #92603

This replaces #113066. I finally came back to this issue and it seems that this approach is still very promising.

As requested, I have added a short explanation as to why CPython module should be moved into a submodule.

cc @JDevlieghere who reviewed on the previous PR earlier.

@llvmbot
Copy link
Member

llvmbot commented Feb 27, 2025

@llvm/pr-subscribers-lldb

Author: wieDasDing (dingxiangfei2009)

Changes

Fix #92603

This replaces #113066. I finally came back to this issue and it seems that this approach is still very promising.

As requested, I have added a short explanation as to why CPython module should be moved into a submodule.

cc @JDevlieghere who reviewed on the previous PR earlier.


Full diff: https://github.com/llvm/llvm-project/pull/129135.diff

2 Files Affected:

  • (modified) lldb/bindings/python/CMakeLists.txt (+6-2)
  • (modified) lldb/bindings/python/python.swig (+6-1)
diff --git a/lldb/bindings/python/CMakeLists.txt b/lldb/bindings/python/CMakeLists.txt
index 69306a384e0b1..c4bf74e094f00 100644
--- a/lldb/bindings/python/CMakeLists.txt
+++ b/lldb/bindings/python/CMakeLists.txt
@@ -59,8 +59,10 @@ endfunction()
 function(finish_swig_python swig_target lldb_python_bindings_dir lldb_python_target_dir)
   # Add a Post-Build Event to copy over Python files and create the symlink to
   # liblldb.so for the Python API(hardlink on Windows).
+  # Note that Swig-generated code is located one level deeper in the `native`
+  # module, in order to avoid cyclic importing.
   add_custom_target(${swig_target} ALL VERBATIM
-    COMMAND ${CMAKE_COMMAND} -E make_directory ${lldb_python_target_dir}
+    COMMAND ${CMAKE_COMMAND} -E make_directory ${lldb_python_target_dir}/native/
     DEPENDS ${lldb_python_bindings_dir}/lldb.py
     COMMENT "Python script sym-linking LLDB Python API")
 
@@ -74,6 +76,8 @@ function(finish_swig_python swig_target lldb_python_bindings_dir lldb_python_tar
       "${LLDB_SOURCE_DIR}/source/Interpreter/embedded_interpreter.py"
       "${lldb_python_target_dir}")
 
+  create_python_package(${swig_target} ${lldb_python_target_dir} "native" FILES)
+
   # Distribute the examples as python packages.
   create_python_package(
     ${swig_target}
@@ -141,7 +145,7 @@ function(finish_swig_python swig_target lldb_python_bindings_dir lldb_python_tar
   endif()
   set(LIBLLDB_SYMLINK_OUTPUT_FILE "_lldb${LLDB_PYTHON_EXT_SUFFIX}")
   create_relative_symlink(${swig_target} ${LIBLLDB_SYMLINK_DEST}
-                          ${lldb_python_target_dir} ${LIBLLDB_SYMLINK_OUTPUT_FILE})
+                          ${lldb_python_target_dir}/native/ ${LIBLLDB_SYMLINK_OUTPUT_FILE})
 
 
   if (NOT WIN32)
diff --git a/lldb/bindings/python/python.swig b/lldb/bindings/python/python.swig
index 278c0eed2bab2..f1b156624e8d8 100644
--- a/lldb/bindings/python/python.swig
+++ b/lldb/bindings/python/python.swig
@@ -50,7 +50,12 @@ Older swig versions will simply ignore this setting.
     import $module
 except ImportError:
     # Relative import should work if we are being loaded by Python.
-    from . import $module"
+    # The cpython module built by swig is pushed one level down into
+    # the native submodule, because at this point the interpreter
+    # is still constructing the lldb module itself.
+    # Simply importing anything using `from . import` constitutes
+    # a cyclic importing.
+    from .native import $module"
 %enddef
 
 // The name of the module to be created.

Copy link
Member

@JDevlieghere JDevlieghere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for adding the comment with the motivation!

@dingxiangfei2009 please let me know if you need me to press the merge button for you.

@dingxiangfei2009
Copy link
Contributor Author

dingxiangfei2009 commented Feb 28, 2025 via email

@JDevlieghere JDevlieghere merged commit 396139a into llvm:main Mar 3, 2025
12 checks passed
@dzhidzhoev
Copy link
Member

@dingxiangfei2009 ,

these changes break the lldb-remote-linux-win buildbot with the following error:

======================================================================
FAIL: test_dwarf (TestArrayFromStdModule.TestCase.test_dwarf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot\as-builder-10\lldb-x-aarch64\llvm-project\lldb\packages\Python\lldbsuite\test\lldbtest.py", line 1784, in test_method
    return attrvalue(self)
           ^^^^^^^^^^^^^^^
  File "C:\buildbot\as-builder-10\lldb-x-aarch64\llvm-project\lldb\packages\Python\lldbsuite\test\decorators.py", line 148, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\buildbot\as-builder-10\lldb-x-aarch64\llvm-project\lldb\test\API\commands\expression\import-std-module\array\TestArrayFromStdModule.py", line 29, in test
    self.expect_expr(
  File "C:\buildbot\as-builder-10\lldb-x-aarch64\llvm-project\lldb\packages\Python\lldbsuite\test\lldbtest.py", line 2540, in expect_expr
    value_check.check_value(self, eval_result, str(eval_result))
  File "C:\buildbot\as-builder-10\lldb-x-aarch64\llvm-project\lldb\packages\Python\lldbsuite\test\lldbtest.py", line 299, in check_value
    test_base.assertSuccess(val.GetError())
  File "C:\buildbot\as-builder-10\lldb-x-aarch64\llvm-project\lldb\packages\Python\lldbsuite\test\lldbtest.py", line 2575, in assertSuccess
    self.fail(self._formatMessage(msg, "'{}' is not success".format(error)))
AssertionError: 'error: While building module 'std' imported from <lldb wrapper prefix>:42:
While building module 'std_core' imported from C:/buildbot/as-builder-10/lldb-x-aarch64/build/include/c++/v1\__algorithm/comp.h:13:
In file included from <module-includes>:2:
In file included from C:/buildbot/as-builder-10/lldb-x-aarch64/build/include/c++/v1\__cstddef/max_align_t.h:13:
C:/buildbot/as-builder-10/lldb-x-aarch64/build/include/c++/v1\stddef.h:38:17: 'stddef.h' file not found
   38 | #  include_next <stddef.h>
      |                 ^~~~~~~~~~

...

error: While building module 'std' imported from <lldb wrapper prefix>:42:
While building module 'std_uchar_h' imported from C:/buildbot/as-builder-10/lldb-x-aarch64/build/include/c++/v1\cuchar:45:
In file included from <module-includes>:1:
In file included from C:/buildbot/as-builder-10/lldb-x-aarch64/build/include/c++/v1\uchar.h:51:
In file included from c:/buildbot/fs/jetson-agx-ubuntu/usr/include\uchar.h:29:
C:/buildbot/as-builder-10/lldb-x-aarch64/build/include/c++/v1\stddef.h:38:17: 'stddef.h' file not found
   38 | #  include_next <stddef.h>
      |                 ^~~~~~~~~~
error: <lldb wrapper prefix>:42:9: could not build module 'std'
   42 | @import std;
      |  ~~~~~~~^~~
' is not success
Config=aarch64-C:\buildbot\as-builder-10\lldb-x-aarch64\build\bin\clang.exe

https://lab.llvm.org/buildbot/#/builders/197/builds/2712/steps/17/logs/FAIL__lldb-api__TestArrayFromStdModule_py

https://lab.llvm.org/buildbot/#/builders/197/builds/2712

Could you take a look at this?

@slydiman
Copy link
Contributor

slydiman commented Mar 4, 2025

It seems this patch have broken a lot of test on the buildbot lldb-remote-linux-win

https://lab.llvm.org/buildbot/#/builders/197/builds/2712

@DavidSpickett
Copy link
Collaborator

Could be a coincidence, but we also have more failures on Windows on Arm - https://lab.llvm.org/buildbot/#/builders/141/builds/6744 that came in with this change.

Might be failing too build the inferior and reporting that badly.

DavidSpickett added a commit that referenced this pull request Mar 4, 2025
DavidSpickett added a commit that referenced this pull request Mar 4, 2025
Reverts #129135 due to buildbot test failures.

Definitely caused remote Linux to Windows failures
(https://lab.llvm.org/buildbot/#/builders/197/builds/2712), may be the
cause of Windows on Arm failures
https://lab.llvm.org/buildbot/#/builders/141/builds/6744.
@DavidSpickett
Copy link
Collaborator

I've reverted this since we have at least 1 confirmed problem.

Also next time please set a valid email address for the contribution so that you get notifications via email as well.

llvm-sync bot pushed a commit to arm/arm-toolchain that referenced this pull request Mar 4, 2025
…" (#129714)

Reverts llvm/llvm-project#129135 due to buildbot test failures.

Definitely caused remote Linux to Windows failures
(https://lab.llvm.org/buildbot/#/builders/197/builds/2712), may be the
cause of Windows on Arm failures
https://lab.llvm.org/buildbot/#/builders/141/builds/6744.
@dingxiangfei2009
Copy link
Contributor Author

Okay!

Thank you for reverting! I will run more test and find a solution.

@dzhidzhoev
Copy link
Member

Okay!

Thank you for reverting! I will run more test and find a solution.

It seems that ScriptInterpreterPython::SharedLibraryDirectoryHelper can't correctly determine path to liblldb.dll after this change.
SharedLibraryDirectoryHelper("build-lldb\Lib\site-packages\lldb\native\_lldb.cp312-win_amd64.pyd") returns build-lldb\Lib\bin\liblldb.dll instead of build-lldb\bin\liblldb.dll.

This should help:

diff --git a/lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp b/lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp
index 00d01981c64f..2dc4ad269172 100644
--- a/lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp
+++ b/lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp
@@ -310,6 +310,7 @@ void ScriptInterpreterPython::SharedLibraryDirectoryHelper(
   // does.
   if (this_file.GetFileNameExtension() == ".pyd") {
     this_file.RemoveLastPathComponent(); // _lldb.pyd or _lldb_d.pyd
+    this_file.RemoveLastPathComponent(); // native
     this_file.RemoveLastPathComponent(); // lldb
     llvm::StringRef libdir = LLDB_PYTHON_RELATIVE_LIBDIR;
     for (auto it = llvm::sys::path::begin(libdir),

jph-13 pushed a commit to jph-13/llvm-project that referenced this pull request Mar 21, 2025
Fix llvm#92603

This replaces llvm#113066. I finally came back to this issue and it seems
that this approach is still very promising.

As requested, I have added a short explanation as to why CPython module
should be moved into a submodule.

cc @JDevlieghere who reviewed on the previous PR earlier.
jph-13 pushed a commit to jph-13/llvm-project that referenced this pull request Mar 21, 2025
)

Reverts llvm#129135 due to buildbot test failures.

Definitely caused remote Linux to Windows failures
(https://lab.llvm.org/buildbot/#/builders/197/builds/2712), may be the
cause of Windows on Arm failures
https://lab.llvm.org/buildbot/#/builders/141/builds/6744.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot run LLDB API Tests on MacOS
6 participants