[vcpkg-get-python-packages] Cache packages via pip precandidate

[WangWeiLin-MV]

WIP

Abstract

Download pypi packages by vcpkg_download_distfile, then install packages offline.

Fix #37473 (review). And any other port that depends on pip install.

Changes

• Bootstrap virtual environment with runtime which has neither standard library venv nor pip (i.e. Embedded Python)
• Add pip command precandidate to obtain the candidate package filename during local installation
• Download packages by vcpkg_download_distfile

WIP

[Neumann-A]

a) this depends on the tool RFC
b) simply no. Doesn't solve the real problem. Simply change vcpkg_get_python_packages so that it is asset cacheable without changing its interface?

[Hoikas]

Python tools need to be the exact same version as the python3 port, or you introduce potential interesting compile failures. AFAIK, the Python 3 port is still on 3.11, but this is using 3.12.

This also seems to be completely unnecessary - we already have the tool package and vcpkg_get_python_packages().

[WangWeiLin-MV]

The purpose of this port is as a module for finding Python interpreter and packages, and implementing #31748 vcpkg-tool-python3-interpreter, and combine and make vcpkg-get-python-packages cacheable.

---

Python tools need to be the exact same version as the python3 port, or you introduce potential interesting compile failures. AFAIK, the Python 3 port is still on 3.11, but this is using 3.12.

The ports/python installs python libraries and binary tools, while current script only find or provide a runtime with packages and is not installed as artifacts. I think this will not affect each other.

[Neumann-A]

Just use the same approach as vcpkg_find_acquire_msys and put it into vcpkg-get-python-packages

[WangWeiLin-MV]

Just use the same approach as vcpkg_find_acquire_msys and put it into vcpkg-get-python-packages

vcpkg-get-python-packages does not provide search function. And no optional virtual environment, it makes difficult to integrate for current port purpose.

About same approach as vcpkg_find_acquire_msys, there are some problems to resolve:

For using pypi source dist requires additional build dependencies, I didn't choose this way.

And for pypi binary dist consists of complex dependencies such as https://pypi.org/pypi/pandas/json requires_dist and lots of wheel https://pypi.org/project/pandas/#files depends on platform. The detection function is used here to obtain dependencies.

[Neumann-A]

@WangWeiLin-MV: I am aware of all what your are telling me. I am still saying use the same approach as vcpkg_find_acquire_msys. Just make it download the wheel and declare the runtime dependencies which can be tested using an from package import * after installation.

YOu also want to automate the generation. So you might want to create a powershell/cmake scripts using
"Invoke-WebRequest -Uri \"https://pypi.python.org/pypi/${package_name}/json/\" -Method GET | ConvertFrom-Json | Select-Object -ExpandProperty info | Select-Object -ExpandProperty version"
you can modify the web request to get the url and filename of the wheel.

[WangWeiLin-MV]

Thanks to @Neumann-A and @Hoikas, I'll continue the discussion later that for the find_python function and whether it replaces vcpkg_find_acquire_program_python and vcpkg-get-python-packages.

Before this, I will first make vcpkg-get-python-packages cacheable.