feature: Python to use uv to resolve and cache dependencies for python runs

[brunorpinho]

Please, allow a flag to resolve dependencies with uv.

My idea is that when env PIP_USE_UV = true worker will download and cache the uv binary and change pip install command in python_executor.rs. Also maps windmills extra index to UV_EXTRA_INDEX_URL.

uv listens to env VIRTUAL_ENV to decide the venv to install it so the executor script can pass that env so that uv targets the correct venv.

There might be some other details that can be discussed later, but it seems to be a promissing way to accelerate python script spin up.

Thank you

References:

https://pypi.org/project/uv/
https://astral.sh/blog/uv

uv pip install --help:

Install packages into the current environment

Usage: uv pip install [OPTIONS] <PACKAGE|--requirement <REQUIREMENT>|--editable <EDITABLE>>

Arguments:
  [PACKAGE]...
          Install all listed packages

Options:
  -r, --requirement <REQUIREMENT>
          Install all packages listed in the given requirements files

  -e, --editable <EDITABLE>
          Install the editable package based on the provided local file path

  -c, --constraint <CONSTRAINT>
          Constrain versions using the given requirements files.
          
          Constraints files are `requirements.txt`-like files that only control the _version_ of a requirement that's installed. However, including a package in a constraints file will _not_ trigger the installation of that package.
          
          This is equivalent to pip's `--constraint` option.

      --override <OVERRIDE>
          Override versions using the given requirements files.
          
          Overrides files are `requirements.txt`-like files that force a specific version of a requirement to be installed, regardless of the requirements declared by any constituent package, and regardless of whether this would be considered an invalid resolution.
          
          While constraints are _additive_, in that they're combined with the requirements of the constituent packages, overrides are _absolute_, in that they completely replace the requirements of the constituent packages.

      --extra <EXTRA>
          Include optional dependencies in the given extra group name; may be provided more than once

      --all-extras
          Include all optional dependencies

  -q, --quiet
          Do not print any output

  -U, --upgrade
          Allow package upgrades

  -v, --verbose
          Use verbose output

  -P, --upgrade-package <UPGRADE_PACKAGE>
          Allow upgrade of a specific package

      --color <COLOR>
          Control colors in output
          
          [default: auto]

          Possible values:
          - auto:   Enables colored output only when the output is going to a terminal or TTY with support
          - always: Enables colored output regardless of the detected environment
          - never:  Disables colored output

      --reinstall
          Reinstall all packages, regardless of whether they're already installed

  -n, --no-cache
          Avoid reading from or writing to the cache
          
          [env: UV_NO_CACHE=]

      --reinstall-package <REINSTALL_PACKAGE>
          Reinstall a specific package, regardless of whether it's already installed

      --cache-dir <CACHE_DIR>
          Path to the cache directory
          
          [env: UV_CACHE_DIR=]

      --offline
          Run offline, i.e., without accessing the network

      --refresh
          Refresh all cached data

      --refresh-package <REFRESH_PACKAGE>
          Refresh cached data for a specific package

      --no-deps
          Ignore package dependencies, instead only installing those packages explicitly listed on the command line or in the requirements files

      --link-mode <LINK_MODE>
          The method to use when installing packages from the global cache
          
          [default: hardlink]

          Possible values:
          - clone:    Clone (i.e., copy-on-write) packages from the wheel into the site packages
          - copy:     Copy packages from the wheel into the site packages
          - hardlink: Hard link packages from the wheel into the site packages

      --resolution <RESOLUTION>
          [default: highest]

          Possible values:
          - highest:       Resolve the highest compatible version of each package
          - lowest:        Resolve the lowest compatible version of each package
          - lowest-direct: Resolve the lowest compatible version of any direct dependencies, and the highest compatible version of any transitive dependencies

      --prerelease <PRERELEASE>
          [default: if-necessary-or-explicit]

          Possible values:
          - disallow:                 Disallow all pre-release versions
          - allow:                    Allow all pre-release versions
          - if-necessary:             Allow pre-release versions if all versions of a package are pre-release
          - explicit:                 Allow pre-release versions for first-party packages with explicit pre-release markers in their version requirements
          - if-necessary-or-explicit: Allow pre-release versions if all versions of a package are pre-release, or if the package has an explicit pre-release marker in its version requirements

  -o, --output-file <OUTPUT_FILE>
          Write the compiled requirements to the given `requirements.txt` file

  -i, --index-url <INDEX_URL>
          The URL of the Python package index (by default: https://pypi.org/simple)
          
          [env: UV_INDEX_URL=]

      --extra-index-url <EXTRA_INDEX_URL>
          Extra URLs of package indexes to use, in addition to `--index-url`
          
          [env: UV_EXTRA_INDEX_URL=]

  -f, --find-links <FIND_LINKS>
          Locations to search for candidate distributions, beyond those found in the indexes.
          
          If a path, the target must be a directory that contains package as wheel files (`.whl`) or source distributions (`.tar.gz` or `.zip`) at the top level.
          
          If a URL, the page must contain a flat list of links to package files.

      --no-index
          Ignore the registry index (e.g., PyPI), instead relying on direct URL dependencies and those discovered via `--find-links`

      --legacy-setup-py
          Use legacy `setuptools` behavior when building source distributions without a `pyproject.toml`

      --no-build
          Don't build source distributions.
          
          When enabled, resolving will not run arbitrary code. The cached wheels of already-built source distributions will be reused, but operations that require building distributions will exit with an error.
          
          Alias for `--only-binary :all:`.

      --no-binary <NO_BINARY>
          Don't install pre-built wheels.
          
          The given packages will be installed from a source distribution. The resolver will still use pre-built wheels for metadata.
          
          Multiple packages may be provided. Disable binaries for all packages with `:all:`. Clear previously specified packages with `:none:`.

      --only-binary <ONLY_BINARY>
          Only use pre-built wheels; don't build source distributions.
          
          When enabled, resolving will not run code from the given packages. The cached wheels of already-built source distributions will be reused, but operations that require building distributions will exit with an error.
          
          Multiple packages may be provided. Disable binaries for all packages with `:all:`. Clear previously specified packages with `:none:`.

  -C, --config-setting <CONFIG_SETTING>
          Settings to pass to the PEP 517 build backend, specified as `KEY=VALUE` pairs

      --strict
          Validate the virtual environment after completing the installation, to detect packages with missing dependencies or other issues

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

[rubenfiszel]

I've investigated using uv but there are quite a few options it didn't support last time I had checked. It would be fine to make it a flag options but right now we do not have the bandwidth to implement and most importantly to test this change as much as this is warranted. If someone want to pick it up as an unstable extra feature, feel free to contribute as a PR

[pyranota]

@brunorpinho We almost replaced pip-compile with uv. And we may replace pip install as well.
#4460