Rewrite Python interpreter discovery

[zanieb]

Updates our Python interpreter discovery to conform to the rules described in #2386, please see that issue for a full description of the behavior. Briefly, we now will search for interpreters that satisfy a requested version without stopping at the first Python executable. Additionally, if retrieving information about an interpreter fails we will continue to search for a working interpreter. We also add the plumbing necessary to request Python implementations other than CPython, though we do not add support for other implementations at this time.

A major internal goal of this work is to prepare for user-facing managed toolchains i.e. fetching a requested version during uv run. These APIs are not introduced, but there is some managed toolchain handling as required for our test suite.

Some noteworthy implementation changes:

• The uv_interpreter::find_python module has been removed in favor of a uv_interpreter::discovery module.
• There are new types to help structure interpreter requests and track sources
• Executable discovery is implemented as a big lazy iterator and is a central authority for source precedence
• uv_interpreter::Error variants were split into scoped types in each module
• There's much more unit test coverage, but not for Windows yet

Remaining work:

• Write new test cases
• Determine correct behavior around executables in the current directory
• Future: Combine PythonVersion and VersionRequest
• Future: Consider splitting ManagedToolchain into local and remote variants
• Future: Add Windows unit test coverage
• Future: Explore behavior around implementation precedence (i.e. CPython over PyPy)

Refactors split into:

• Move py launcher handling into separate module #3329
• Update toolchain discovery to avoid runtime panic #3330
• Split virtual environment detection into a dedicated module #3331
• Separate interpreter selectors into implementation and platform modules #3332

Closes #2386

[codspeed-hq]

CodSpeed Performance Report

Merging #3266 will not alter performance

_{Comparing zb/interp-request-ii (1b212ab) with main (dfd6ccf)}

Summary

✅ 12 untouched benchmarks

[zanieb]

Example test setup

gh pr checkout 3266
cargo build
alias uv=$(pwd)/target/debug/uv
uv version

I'll do some local QA and share helpful commands here

[charliermarsh]

Not sure if intentional, but I have pypy (which is PyPy at Python 2.7) and pypy39 (which is PyPy at Python 3.9) in path, and cargo run venv --python pypy fails because it tries to use the one called pypy:

puffin on  zb/interp-request-ii [$] is 📦 v0.1.44 via 🐍 v3.11.8 via 🦀 v1.78.0 took 2s
❯ cargo run venv --python pypy
   Compiling uv v0.1.44 (/Users/crmarsh/workspace/puffin/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.75s
     Running `target/debug/uv venv --python pypy`
  × Can't use Python at `/opt/homebrew/bin/pypy`
  ╰─▶ Python executable does not support `-I` flag. Please use Python 3.8 or newer.

puffin on  zb/interp-request-ii [$] is 📦 v0.1.44 via 🐍 v3.11.8 via 🦀 v1.78.0 took 2s
❯ cargo run venv --python pypy3.9
   Compiling uv v0.1.44 (/Users/crmarsh/workspace/puffin/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.02s
     Running `target/debug/uv venv --python pypy3.9`
Using Python 3.9.18 interpreter at: /opt/homebrew/bin/pypy3.9
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate

Should it resolve to the pypy3.9?

cargo run venv --python pypy3.9 does work as expected.

[charliermarsh]

Otherwise, my own smoke-testing works as expected.

[charliermarsh]

I would personally expect this to work -- I copied a Python to foo locally, and --python ./foo discovers it but --python foo does not:

puffin on  zb/interp-request-ii [$?]
❯ cargo run venv --python foo
   Compiling uv v0.1.44 (/Users/crmarsh/workspace/puffin/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.87s
     Running `target/debug/uv venv --python foo`
  × Requested Python executable `foo` not found in PATH

puffin on  zb/interp-request-ii [$?]
❯ cargo run venv --python ./foo
   Compiling uv v0.1.44 (/Users/crmarsh/workspace/puffin/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.72s
     Running `target/debug/uv venv --python ./foo`
Using Python 3.9.18 interpreter at: foo
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate

I think we fixed this in the prior implementation and that it was considered a bug.

[zanieb]

I think your pypy example fails because we don't have support for detecting that as an implementation name yet (that will happen in a follow up pull request). Here, cpython is the only implementation name variant (just to reduce scope).