[pybind11] Fail to link debug python library

[zyrkiel]

Describe the bug
While linking to a custom library depending on pybind11 in Debug I get an undefined reference. It is due to the fact that the release Python3 lib is given to the link instead of the debug one.
This issue might be related to #33724

Environment

• OS: Ubuntu 22.04
• Compiler: gcc 13 or clang 17
• Genrator: Unix Makefiles or Ninja
• Python3.11 and Python3.11d are installed locally but I would like using vcpkg python lib
• Config: Debug

To Reproduce
Code is available here: https://github.com/zyrkiel/test_link_python_debug

Steps to reproduce the behavior:

1. git clone --recurse-submodules https://github.com/zyrkiel/test_link_python_debug.git
2. cd test_link_python_debug
3. cmake -DBUILD_SHARED_LIBS:BOOL=OFF -DCMAKE_TOOLCHAIN_FILE=vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_MANIFEST_INSTALL:BOOL=ON -DVCPKG_HOST_TRIPLET=x64-linux -DVCPKG_TARGET_TRIPLET=x64-linux -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE:STRING=Debug -DCMAKE_INSTALL_PREFIX:STRING=install -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -S . -B build
4. cmake --build build --config Debug

However, running again the configuration and the build will lead to a success.

Expected behavior
cmake should find the debug library of python and links it

Failure logs

First Configure output:

-- Found pybind11: /home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/include (found version "2.11.1")
-- Found Python3: /home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/tools/python3/python3.11 (found version "3.11.5") found components: Interpreter Development Development.Module Development.Embed

First Build output:

[...]
[100%] Linking CXX executable PythonLinkTest
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/PythonLinkTest.dir/link.txt --verbose=1
/usr/bin/c++ -g CMakeFiles/PythonLinkTest.dir/main.cpp.o -o PythonLinkTest  **foo_lib/libFoo.a vcpkg_installed/x64-linux/lib/libpython3.11.a** -ldl 
/usr/bin/ld: foo_lib/libFoo.a(foo.cpp.o): in function `Py_INCREF':
/home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/include/python3.11/object.h:500: undefined reference to `_Py_RefTotal'
/usr/bin/ld: /home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/include/python3.11/object.h:500: undefined reference to `_Py_RefTotal'
/usr/bin/ld: foo_lib/libFoo.a(foo.cpp.o): in function `Py_DECREF':
/home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/include/python3.11/object.h:520: undefined reference to `_Py_RefTotal'
/usr/bin/ld: /home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/include/python3.11/object.h:520: undefined reference to `_Py_RefTotal'
/usr/bin/ld: /home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/include/python3.11/object.h:523: undefined reference to `_Py_NegativeRefcount'
[...]

Second Configure output:

-- Found pybind11: /home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/include (found version "2.11.1")
-- Found Python3: /home/tgeffroy/GitRepo/git/test_link_python_debug/build/vcpkg_installed/x64-linux/tools/python3/python3.11 (found version "3.11.5") found components: Interpreter Development Development.Module Development.Embed

Second Build output:

[ 75%] Linking CXX executable PythonLinkTest
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/PythonLinkTest.dir/link.txt --verbose=1
/usr/bin/c++ -g CMakeFiles/PythonLinkTest.dir/main.cpp.o -o PythonLinkTest  foo_lib/libFoo.a vcpkg_installed/x64-linux/debug/lib/libpython3.11d.a -ldl 
gmake[2]: Leaving directory '/home/tgeffroy/GitRepo/git/test_link_python_debug/build'
[100%] Built target PythonLinkTest

Additional context
The issue is not dependent from compiler, nor generator.
As you can see, the link is trying to use the release lib foo_lib/libFoo.a vcpkg_installed/x64-linux/lib/libpython3.11.a

The proposed solution to #33724 isn't working: set(Python_FIND_ABI "ON" "ANY" "ANY") before trying to find packages. It gives the same behavior.

[zyrkiel]

It seems more logic to find python3 before pybind11.
So I have switched the 2 lines.

After that, I have realized that I should use Python3_FIND_ABI instead of Python_FIND_ABI since I am trying to find Python3.

Finally, removing Python3 from the vcpkg.json dependencies enables the configure to end successfully and to link to local Python.

You can find the workaround.patch in the repository.

However, I still would like to use vcpkg python3 package and not to specify the build type at the configure phase.

[github-actions]

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 180 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

[zyrkiel]

Keep open