Custom sources for python distribution binaries

[brendan-morin]

Many companies have policies around sourcing artifacts from internal sources only. I'm seeing in the docs that uv uses pre-built third-party distributions from the python-build-standalone project. Is it possible to configure uv to point to an alternate private index for python version distributions?

I imagine as long as there's a well-defined API for how uv queries and returns binaries, the actual source from which these are downloaded should probably not matter much?

Apologies if this is a duplicate, I searched around but didn't spot an existing issue.

[Crypto-Spartan]

Timing is wild, I was just about to submit an issue with this same request. I would love support for changing the URL that uv uses for it's request to the python-build-standalone project as well. (also due to a corporate environment)

[zanieb]

Yep! UV_PYTHON_INSTAL_MIRROR

[zanieb]

Described at https://docs.astral.sh/uv/configuration/environment/

[Crypto-Spartan]

Thank you for the quick response!

Additionally, is there support for providing uv a filepath instead of a URL to the .tar.gz that's downloaded from the python-build-standalone project (cpython-3.X.X+<date>-<target>-install_only_stripped.tar.gz)?

[zanieb]

You can use file:// in the install mirror.

[Crypto-Spartan]

so I'm currently trying to install https://github.com/indygreg/python-build-standalone/releases/download/20241008/cpython-3.12.7+20241008-x86_64_v3-unknown-linux-gnu-pgo+lto-full.tar.zst with uv.

I have it downloaded in $HOME/downloads/python_standalone/20241008/cpython-3.12.7+20241008-x86_64_v3-unknown-linux-gnu-pgo+lto-full.tar.zst

I set UV_PYTHON_INSTALL_MIRROR='file:///home/<user>/downloads/python_standalone'

When I attempt to run uv python install 3.12.7, it fails with

Searching for Python versions matching: Python 3.12.7
error: No download found for request: cpython-3.12.7-linux-x86_64-gnu

> echo $UV_PYTHON_INSTALL_MIRROR
file:///home/<user>/downloads/python_standalone

Could you provide help/guidance as to how I should proceed? Thanks for your help

I also tried with UV_PYTHON_INSTALL_MIRROR='file://home/<user>/downloads/python_standalone' just incase I had too many slashes after the file:, still didn't work

I'm on Ubuntu 22.04 LTS (jammy) if that matters

[zanieb]

Can you share verbose logs? i.e. RUST_LOG=uv=trace uv python install -v 3.12.7

Your uv version needs to support that managed download, so you might just be on a uv version that only has metadata for 3.12.6 and not 3.12.7 yet?

[Crypto-Spartan]

> RUST_LOG=uv-trace uv python install -v 3.12.7
DEBUG uv 0.4.18
TRACE Checking lock for `/home/<user>/.local/share/uv/python` at `/home/<user>/.local/share/uv/python/.lock`
DEBUG Acquired lock for `/home/<user>/.local/share/uv/python`
Searching for Python versions matching: Python 3.12.7
TRACE ld path: /lib64/ld-linux-x86-64.so.2
TRACE stdout output from `ld`: ""
TRACE stderr output from `ld`: "/lib64/ld-linux-x86-64.so.2: missing program name\nTry '/lib64/ld-linux-x86-64.so.2 --help for more information.\n"
TRACE stdout ouput from `ldd --version`: ld.so (Ubuntu GLIBC 2.35-0ubuntu3.8) stable release version 2.35.\nCopyright ...
TRACE Found manylinux 2.35 in stdout of `ldd --version`
DEBUG Released lock at `/home/<user>/.local/share/uv/python/.lock`
error: No download found for request: cpython-3.12.7-linux-x86_64-gnu

uv 0.4.18 is the latest version of uv I'm able to grab off of the pypi mirror in my corporate environment

[zanieb]

3.12.7 was added in #7880 which is in uv 0.4.19

[Crypto-Spartan]

Tried again with https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-pgo+lto-full.tar.zst instead this time.

> uv python install 3.12.6
Searching for Python versions matching: Python 3.12.6
cpython-3.12.6-linux-x86_64-gnu
error: Failed to install cpython-3.12.6-linux-x86_64-gnu
  Caused by: Failed to extract archive: cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
  Caused by: Invalid gzip header
cpython-3.12.6-linux-x86_64-gnu

[Crypto-Spartan]

does uv only install the x86_64-unknown-linux-gnu-install_only_stripped.tar.gz?
Why can't I install the x86_64-unknown-linux-gnu-pgo+lto-full.tar.zst?

[zanieb]

Sorry this isn't well documented — yes, we only support the target defined in our download metadata.

We could probably support more in the future, it's just more complicated.

[zanieb]

Basically the reason you can't, is that we generate JSON metadata from GitHub release

uv/crates/uv-python/download-metadata.json

Lines 398 to 409 in f6fd849

	"cpython-3.12.7-darwin-aarch64-none": {
	"name": "cpython",
	"arch": "aarch64",
	"os": "darwin",
	"libc": "none",
	"major": 3,
	"minor": 12,
	"patch": 7,
	"prerelease": "",
	"url": "https://github.com/indygreg/python-build-standalone/releases/download/20241008/cpython-3.12.7%2B20241008-aarch64-apple-darwin-install_only_stripped.tar.gz",
	"sha256": "dc1f4d80c9b4ae40ba2c725e310e61446f8f778a7bea4efa56e4b1b4446e0a1a"
	},

then encode it as static Rust code for performance

uv/crates/uv-python/src/downloads.inc

Lines 471 to 484 in f6fd849

	ManagedPythonDownload {
	key: PythonInstallationKey {
	major: 3,
	minor: 12,
	patch: 7,
	prerelease: Cow::Borrowed(""),
	implementation: LenientImplementationName::Known(ImplementationName::CPython),
	arch: Arch(target_lexicon::Architecture::Aarch64(target_lexicon::Aarch64Architecture::Aarch64)),
	os: Os(target_lexicon::OperatingSystem::Darwin),
	libc: Libc::None,
	},
	url: "https://github.com/indygreg/python-build-standalone/releases/download/20241008/cpython-3.12.7%2B20241008-aarch64-apple-darwin-install_only_stripped.tar.gz",
	sha256: Some("dc1f4d80c9b4ae40ba2c725e310e61446f8f778a7bea4efa56e4b1b4446e0a1a")
	},

then when you ask for a custom mirror we transform the static URL

uv/crates/uv-python/src/downloads.rs

Lines 577 to 585 in 891c91d

	if let Ok(mirror) = std::env::var("UV_PYTHON_INSTALL_MIRROR") {
	let Some(suffix) = self.url.strip_prefix(
	"https://github.com/indygreg/python-build-standalone/releases/download/",
	) else {
	return Err(Error::Mirror("UV_PYTHON_INSTALL_MIRROR", self.url));
	};
	return Ok(Url::parse(
	format!("{}/{}", mirror.trim_end_matches('/'), suffix).as_str(),
	)?);

This is quite a naive implementation, but it's intended to unblock core functionality not be fully feature complete.

[Crypto-Spartan]

Is there any specific reason why the generated json only includes the install_only_stripped.tar.gz and none of the other python standalone builds? I'm willing to work on a PR to support the other builds if you're open to accepting it.

[zanieb]

I think we just adopted the "choose a single build" behavior from Rye's code. I'd love to support other builds (see #8019), though it's probably non-trivial. If you want to work on it, I'd appreciate it. Otherwise I'll probably work on #8019 in the near future.

[Crypto-Spartan]

something weird is happening still, it looks like uv is attempting to download from github directly still which won't be possible in my environment.

cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz is currently in /home/<user>/downloads/python_standalone

> echo $UV_PYTHON_INSTALL_MIRROR
file:///home/<user>/downloads/python_standalone

> RUST_LOG=uv-trace uv python install -v 3.12.6
DEBUG uv 0.4.18
TRACE Checking lock for `/home/<user>/.local/share/uv/python` at `/home/<user>/.local/share/uv/python/.lock`
DEBUG Acquired lock for `/home/<user>/.local/share/uv/python`
Searching for Python versions matching: Python 3.12.6
TRACE ld path: /lib64/ld-linux-x86-64.so.2
TRACE stdout output from `ld`: ""
TRACE stderr output from `ld`: "/lib64/ld-linux-x86-64.so.2: missing program name\nTry '/lib64/ld-linux-x86-64.so.2 --help for more information.\n"
TRACE Tried to find musl version by running `"/lib64/ld-linux-x86-64.so.2"`, but failed: Could not find musl version in output of: `/lib64/ld-linux-x86-64.so.2`
TRACE Tried to find libc version from possible symlink at "/lib64/ld-linux-x86-64.so.2", but failed: Failed to find glibc version in the filename of linker: `/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2`
TRACE stdout ouput from `ldd --version`: ld.so (Ubuntu GLIBC 2.35-0ubuntu3.8) stable release version 2.35.\nCopyright ...
TRACE Found manylinux 2.35 in stdout of `ldd --version`
DEBUG Using request timeout of 30s
TRACE Handling request for https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
TRACE Request for https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz is unauthenticated, checking cache
TRACE No credentials in cache for URL https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
TRACE Attempting unauthenticated request for https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
DEBUG Downloading https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz to temporary location: /home/<user>/.local/share/uv/python/.cache/.tmpYs2Aii
DEBUG Extracting cpython-3.12.6%2B20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
error: Failed to install cpython-3.12.6-linux-x86_64-gnu
  Caused by: Failed to extract archive: cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
  Caused by: Invalid gzip header
DEBUG Released lock at `/home/<user>/.local/share/uv/python/.lock`

[zanieb]

On my machine, I get something like this if I specify an arbitrary directory:

❯ UV_PYTHON_INSTALL_MIRROR="file:///Users/zb/test" RUST_LOG=uv=trace uv python install -v 3.12.6
DEBUG uv 0.4.18
TRACE Checking lock for `/Users/zb/.local/share/uv/python` at `/Users/zb/.local/share/uv/python/.lock`
DEBUG Acquired lock for `/Users/zb/.local/share/uv/python`
Searching for Python versions matching: Python 3.12.6
DEBUG Using request timeout of 30s
error: Failed to install cpython-3.12.6-macos-aarch64-none
  Caused by: failed to query metadata of file `/Users/zb/test/20240909/cpython-3.12.6+20240909-aarch64-apple-darwin-install_only_stripped.tar.gz`
  Caused by: No such file or directory (os error 2)
DEBUG Released lock at `/Users/zb/.local/share/uv/python/.lock`

[zanieb]

Similarly:

❯ wget https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6%2B20240909-aarch64-apple-darwin-install_only_stripped.tar.gz
❯ mkdir 20240909
❯ mv cpython-3.12.6+20240909-aarch64-apple-darwin-install_only_stripped.tar.gz 20240909
❯ UV_PYTHON_INSTALL_MIRROR="file:///Users/zb/workspace/uv" RUST_LOG=uv=trace uv python install -v 3.12.6
DEBUG uv 0.4.18
TRACE Checking lock for `/Users/zb/.local/share/uv/python` at `/Users/zb/.local/share/uv/python/.lock`
DEBUG Acquired lock for `/Users/zb/.local/share/uv/python`
Searching for Python versions matching: Python 3.12.6
DEBUG Using request timeout of 30s
DEBUG Downloading file:///Users/zb/workspace/uv/20240909/cpython-3.12.6%2B20240909-aarch64-apple-darwin-install_only_stripped.tar.gz to temporary location: /Users/zb/.local/share/uv/python/.cache/.tmpvg3LcW
DEBUG Extracting cpython-3.12.6%2B20240909-aarch64-apple-darwin-install_only_stripped.tar.gz
DEBUG Moving /Users/zb/.local/share/uv/python/.cache/.tmpvg3LcW/python to /Users/zb/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none
Installed Python 3.12.6 in 380ms
 + cpython-3.12.6-macos-aarch64-none
DEBUG Released lock at `/Users/zb/.local/share/uv/python/.lock`

[Crypto-Spartan]

Figured it out

my filename was urlencoded so was actually cpython-3.12.6%2B20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
instead of cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
(%2B vs +)

Thanks for all of your help

[zanieb]

That makes sense. We can add more documentation around this.

[guillemc23]

I'm a bit confused. We're currently migrating to uv and we need to install python packages from private sources.

Currently, I'm just adding the following to my pyproject.toml:

[tool.uv]
index-url = "https://<user>:<password>@<host>"

Which is working properly.

I have been unable to find a way to configure this authentication in order to avoid just hardcoding the credentials for obvious reasons. Is this somehow related to this issue? Also, is this the way I should be doing this?

EDIT: I believe what I'm referring to is this PR #7741

[chrisrodrigue]

Just took a look at #7741 and I am wondering why the environment variables can't be specified in the pyproject.toml for the index URLs.

[zanieb]

@guillemc23 sorry this issue is about downloading Python distributions — not index URLs for packages. Could you open a new issue for your question instead?

[MeitarR]

Figured it out

my filename was urlencoded so was actually cpython-3.12.6%2B20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz instead of cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz (%2B vs +)

Thanks for all of your help

Opened PR with a create mirror script to make it easier #8548

[MeitarR]

maybe relevant #10203