Expand environment variables in user-provided index URLs

[charliermarsh]

Right now, we only support expansion (1) by the shell (of course) and (2) in requirements files. But the following don't expand:

• cargo run pip install flask --index-url 'https://${PYPI_USERNAME}.com/repository/pypi-proxy/simple/flask/'
• Anything in a uv.toml or pyproject.toml file

[charliermarsh]

It looks like pip does not support either of these.

[charliermarsh]

Poetry does not support the latter: python-poetry/poetry#208

[charliermarsh]

Rust does not support this but the issue is not closed: rust-lang/cargo#10789

[charliermarsh]

I am not strictly opposed to it but we should get some more opinions (\cc @zanieb @BurntSushi @konstin).

[chrisrodrigue]

PDM supports this: Store credentials with the index, Local dependencies

[chrisrodrigue]

Hatch also supports this: Context formatting

[eth3lbert]

I looked around Poetry's document and found this https://python-poetry.org/docs/configuration/#http-basicnameusernamepassword .

[CoolCat467]

I know I'm not a maintainer of uv, but I strongly suggest that if environment lookups were added that they be opt-in. I've heard of things going very bad security wise from environment variables.

[zanieb]

Can you share more details or examples please @CoolCat467?

[zanieb]

Something like --allow-variable <NAME> might make sense for opt-in per-variable. We probably want to consider how we want the UX to work for lockfiles and credentials in general. I think expanding environment variables is probably not the best solution, it seems like we'd be better off finding a matching URL in our configuration file and pulling authentication from there (we need and want to support declaring indexes anyway).

[eth3lbert]

Cargo has https://doc.rust-lang.org/cargo/reference/config.html#env

Might not directly related but there's also https://doc.rust-lang.org/cargo/reference/config.html#credentials.

[chrisrodrigue]

Related: #5119

[chrisrodrigue]

@CoolCat467

I know I'm not a maintainer of uv, but I strongly suggest that if environment lookups were added that they be opt-in. I've heard of things going very bad security wise from environment variables.

Currently uv.lock exposes API keys in source URLs which means a lot of users can’t check it into version control.

One of the strengths of environment variables is avoiding version control. Most VCSs support masking environment variables from job/log output when expanded. See GitLab and GitHub docs.

They aren’t 100% foolproof. I think there’s a tradeoff between flexibility and security.

[BurntSushi]

cargo run pip install flask --index-url 'https://${PYPI_USERNAME}.com/repository/pypi-proxy/simple/flask/'

I think I would personally find it very surprising if an environment variable got expanded here. Basically, if I'm running a Unix shell command and I use single quotes, I have a very strong prior that the string is interpreted literally with no interpolation.

Something like --allow-variable <NAME> might make sense for opt-in per-variable. We probably want to consider how we want the UX to work for lockfiles and credentials in general. I think expanding environment variables is probably not the best solution, it seems like we'd be better off finding a matching URL in our configuration file and pulling authentication from there (we need and want to support declaring indexes anyway).

👍 on this (including that env vars are maybe not the best solution to this specific problem), but having an explicit --allow-variable I think would help mitigate some concerns here.

[jfgordon2]

I'm currently looking at using uv instead of pipenv for our projects, and env var expansion is something we use with pipenv and our AWS CodeArtifact repo. Not having this makes things a bit more complicated to use uv (i.e. we can't easily specify in the pyproject.toml file where the packages are located without constant updates since the index url contains the auth and it's only good for a limited time).

I'm not sure I understand the concern around env var expansion within a url for this use-case, though.