build: Incorrect python interpreter found by python.cmake on Nix #66236
Comments
teburd
changed the title
|
Diff of my cmake changes to print out a few variables that helped me... I couldn't attach as a file because github hates patches |
|
@teburd thanks a lot for this useful information. I already now have some idea of what might be happening, but need to do some testing on this, but unfortunately not having time atm for further investigation. Marked as |
It's documented here:
and exactly allows you to control behavior so that CMake looks up at those paths before the default locations. |
that’s not the way cmakes own find python module works though, it ignores the CMake environment variables. So this is confusing |
oh, then I misread your comment. I thought that if you defined |
|
See https://gitlab.kitware.com/cmake/cmake/-/blob/master/Modules/FindPython/Support.cmake#L1786 to start, the very first branch for virtualenvs ignores the cmake env vars. cmake's FindPython module works well for me, but I can understand if there's confusing things that happen when it goes looking around registries and such for the latest python install. @tejlmand did you try setting the strategy for FindPython? https://gitlab.kitware.com/cmake/cmake/-/blob/master/Modules/FindPython.cmake#L314 seems like using the LOCATION strategy might be just fine? |
Unfortunately its not. Been there, tried that. The location still doesn't guarantee the order in which Python installations are tried (those in path which #24364 fixes). |
|
@tejlmand Setting Python3_EXECUTABLE doesn't seem to be possible when running twister unless I've missed something (very possible!). I'm carrying around a patch to do my work now which is kind of painful. It seems like in retrospect all the issues here are related to windows and python, can we use the cmake python module on all platforms but windows perhaps? |
Ok I figured it out, maybe somewhat amusingly... I do
|
I think the "bigger" question is: if the real-world issues are (as expected) very different between Windows and Linux, then why is 100% of the workaround code shared between the two? Zephyr's C code is (correctly) split between "arch/" code for arch-specific problems versus common code for common problems. So why does PS: we haven't even looked at macOS/brew yet... |
|
Related, and probably better described in issue #70258 |
Describe the bug
TL;DR: CMake's FindPython module has 3kloc to handle many edge cases, and carefully does things with find_program that we are not doing. I happened to find an edge case seemingly.
I'd expect python.cmake to choose the first python in my path, it instead seems to choose somewhat at random a python interpreter on my path.
find_package(Python COMPONENT INTERPRETER) seems to do this just fine, the current find_program() usage does not.
I've added some message() outputs in python.cmake and west.cmake to try and provide some clues, diff attached
output from this...
I'd note that which python points at my virtual environment, which itself holds a static link to a nix install of python with various python packages (nix will generate a virtualenv like wrapper itself), and that links to a real python interpreter. None of those are actually selected though. Instead it appears that, at perhaps random or some undefined ordering, another python in my PATH var is selected. That's seemingly, to me, the wrong behavior and very unexpected.
If I replace all of python.cmake with a simple find_package(Python COMPONENT INTERPRETER) as seen in #65995 the symlinked python in my virtualenv is found correctly.
Some other useful environment variables perhaps give some clues
nix will set PATH, CMAKE_, and other environment variables when being used to enable builds with its unique filesystem layout.
env | grep python3-3.10.13 shows... (sorry for the large text blob here)
If you look carefully PATH shows my virtualenv as the first path in the list, /home/tburdick/z/env/bin
BUT
The found cmake interpreter is the first thing in CMAKE_LIBRARY_PATH, CMAKE_PREFIX_PATH, CMAKE_INLCUDE_PATH
Perhaps find_program is somehow looking at those paths first despite my PATH, the thing I actually care about, having the virtualenv correctly there? So I tried and I found...
So to me, it appears like CMAKE_PREFIX_PATH is taking precedence over my shell path for find_program. It seems this is just the expected behavior of CMake which... seems incredibly wrong and unintuitive.
So much so that FindPython.cmake finds the correct python! It does that by...
https://gitlab.kitware.com/cmake/cmake/-/blob/master/Modules/FindPython/Support.cmake#L1846
Avoiding all the cmake environment variables, ignoring all the cmake paths...
So we have a bug in our python.cmake that does not exist in cmake's own FindPython support package.
To Reproduce
I suppose try adding a bogus python install to CMAKE_PREFIX_PATH and watch everything fail oddly.
Expected behavior
The first python interpreter in my PATH var is used.
Impact
Building docs is broken for me
The text was updated successfully, but these errors were encountered: