Python detection macro in cmake fails to detect highest installed version

[nashif]

Describe the bug

Python detection fails to detect highest installed version, instead it find a compatibility and older version installed.

To Reproduce

Including boilerplate (Zephyr base): /home/nashif/Work/zephyrproject/zephyr/cmake/app/boilerplate.cmake
-- Application: /home/nashif/Work/zephyrproject/zephyr/samples/hello_world
-- Zephyr version: 2.2.99 (/home/nashif/Work/zephyrproject/zephyr)
-- Found PythonInterp: /usr/bin/python3.6 (found suitable version "3.6.10", minimum required is "3.6")
-- Board: nsim_em
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'west.version'
CMake Error at /home/nashif/Work/zephyrproject/zephyr/cmake/host-tools.cmake:26 (message):
  Unable to import west.version from '/usr/bin/python3.6'
Call Stack (most recent call first):
  /home/nashif/Work/zephyrproject/zephyr/cmake/app/boilerplate.cmake:483 (include)
  /home/nashif/Work/zephyrproject/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
  /home/nashif/Work/zephyrproject/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:35 (include_boilerplate)
  CMakeLists.txt:5 (find_package)

However, I have 3.7.x

i9:zephyr(master): python3 --version
Python 3.7.6

Expected behavior

cmake should find the highest version installed.

Impact

Failure to build

Additional context

Looking at cmake/python.cmake

# The 'FindPythonInterp' that is distributed with CMake 3.8 has a bug
# that we need to work around until we upgrade to 3.13. Until then we
# maintain a patched copy in our repo. Bug:
# https://github.com/zephyrproject-rtos/zephyr/issues/11103
set(PythonInterp_FIND_VERSION 3.6)
set(PythonInterp_FIND_VERSION_COUNT 2)
set(PythonInterp_FIND_VERSION_MAJOR 3)
set(PythonInterp_FIND_VERSION_MINOR 6)
set(PythonInterp_FIND_VERSION_EXACT 0)
set(PythonInterp_FIND_REQUIRED 1)
include(${ZEPHYR_BASE}/cmake/backports/FindPythonInterp.cmake)

That text seems to be outdated and needs to be updated. we already use 3.13, is all of this still needed?

If I make this change:

diff --git a/cmake/python.cmake b/cmake/python.cmake
index 156bbc921c..cdd1025d2a 100644
--- a/cmake/python.cmake
+++ b/cmake/python.cmake
@@ -16,7 +16,7 @@ endif()
 set(PythonInterp_FIND_VERSION 3.6)
 set(PythonInterp_FIND_VERSION_COUNT 2)
 set(PythonInterp_FIND_VERSION_MAJOR 3)
-set(PythonInterp_FIND_VERSION_MINOR 6)
-set(PythonInterp_FIND_VERSION_EXACT 0)
+#set(PythonInterp_FIND_VERSION_MINOR 6)
+#set(PythonInterp_FIND_VERSION_EXACT 0)
 set(PythonInterp_FIND_REQUIRED 1)
 include(${ZEPHYR_BASE}/cmake/backports/FindPythonInterp.cmake)

I get the expected behavior:

-- west build: generating a build system
Including boilerplate (Zephyr base): /home/nashif/Work/zephyrproject/zephyr/cmake/app/boilerplate.cmake
-- Application: /home/nashif/Work/zephyrproject/zephyr/samples/hello_world
-- Zephyr version: 2.2.99 (/home/nashif/Work/zephyrproject/zephyr)
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.7.6", minimum required is "3.6")
-- Board: nsim_em
-- Found west: /home/nashif/.local/bin/west (found suitable version "0.7.2", minimum required is "0.7.1")
-- Found toolchain: zephyr (/home/nashif/Work/SDK/0.11.2)
-- Found BOARD.dts: /home/nashif/Work/zephyrproject/zephyr/boards/arc/nsim/nsim_em.dts
-- Generated zephyr.dts: /home/nashif/Work/zephyrproject/zephyr/build/zephyr/zephyr.dts

[mbolivar-nordic]

That text seems to be outdated and needs to be updated. we already use 3.13, is all of this still needed?

I think no. This backport is not necessary. And in fact FindPythonInterp is deprecated since CMake 3.12:

https://cmake.org/cmake/help/v3.12/module/FindPythonInterp.html

We should be using this instead:

https://cmake.org/cmake/help/v3.13/module/FindPython3.html#module:FindPython3

[tejlmand]

@nashif already looking at this for another reason, see this: #23981 (comment)

We might be able to completely remove Zephyr local copy for finding ZephyrInterp

[tejlmand]

Seems this bug has been re-introduced #11103 by this commit a5eb70f 😞

-- Zephyr version: 2.2.99 (C:/projects/github/ncs/zephyr)
CMake Error at C:/Program Files/CMake/share/cmake-3.14/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find PythonInterp: Found unsuitable version "2.7.9", but required
  is at least "3.6" (found C:/Python27/python.exe)
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.14/Modules/FindPackageHandleStandardArgs.cmake:376 (_FPHSA_FAILURE_MESSAGE)
  C:/projects/github/ncs/zephyr/cmake/backports/FindPythonInterp.cmake:174 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  C:/projects/github/ncs/zephyr/cmake/python.cmake:22 (include)
  C:/projects/github/ncs/zephyr/cmake/app/boilerplate.cmake:115 (include)
  C:/projects/github/ncs/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
  C:/projects/github/ncs/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:35 (include_boilerplate)
  CMakeLists.txt:5 (find_package)

going to ensure #11103 is fixed again, and check the following issues are still fixed when switching to CMake built-in Python search.
#12066
#20303

[tejlmand]

Expected behavior

cmake should find the highest version installed.

@nashif actually, the current design is made so it should choose the version first in path, which was even an issue raised by you, #11857
PR: #11907.

In my opinion, CMake should choose:

1. The python 3.x in path that meets the minimum required Python3 version
   a. If multiple Python 3.x is in path, choose the highest, for example
   - /usr/bin/python3
   - /usr/bin/python3.6
   - /usr/bin/python3.7
   - /usr/bin/python3.8
   Should choose 3.8 (Note, this is not applicable on windows, where python.exe is the name for all versions)

2. In no Python 3.x in path, then select highest installed version that is detected