Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use users pip.conf #1404

Open
msharp9 opened this issue Feb 16, 2024 · 45 comments
Open

Use users pip.conf #1404

msharp9 opened this issue Feb 16, 2024 · 45 comments
Labels
compatibility Compatibility with a specification or another tool configuration Settings and such

Comments

@msharp9
Copy link

msharp9 commented Feb 16, 2024

Pip allows a user to set configuration settings inside a pip.conf, usually stored at ~/.config/pip/pip.conf. For example, setting a default index-url. uv seems to ignore these settings completely.

This is a problem since enterprise companies typically have their own PyPI images and don't want users to use the public PyPI for security reasons. For example, users who use AWS codeartifact have this config file updated automatically when they run aws login. https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-pip.html

@zanieb zanieb added enhancement New feature or improvement to existing functionality compatibility Compatibility with a specification or another tool labels Feb 16, 2024
@zanieb
Copy link
Member

zanieb commented Feb 16, 2024

Thanks for the clear issue!

@zanieb zanieb added configuration Settings and such and removed enhancement New feature or improvement to existing functionality labels Feb 16, 2024
@owenlamont
Copy link

I was just about to raise an issue for this exact use case. We also use codeartifact for a few in house packages but mostly pull public PyPi packages via AWS codeartifact.

Anyway +1 to this. Would love to use uv if this can be supported.

@kkpattern
Copy link

We also set index-url in our venv by pip config --site set install.index-url "http://path/to/internal/pypi/server"

@schlamar
Copy link

BTW, for Windows this is %APPDATA%\pip\pip.ini, see https://pip.pypa.io/en/stable/topics/configuration/#location

@mcrumiller
Copy link

Can you perhaps add to the README instructions on how to set the index-url? My company uses a proxy repo and it's not obvoius how to read from it. Do we have to supply --index-url every time we call python -m pip uv? That seems a bit cumbersome.

@zanieb
Copy link
Member

zanieb commented Feb 25, 2024

Hi @mcrumiller you can use UV_INDEX_URL instead. We may want to add all of our environment variables to the README, but they are currently listed in the --help output e.g.

  -i, --index-url <INDEX_URL>
          The URL of the Python package index (by default: https://pypi.org/simple)
          
          [env: UV_INDEX_URL=]

@PetterS
Copy link

PetterS commented Mar 1, 2024

The environment variables are nice, but reading the pip config file would be very good for enterprises. It would allow a seamless transition to uv without installing additional configuration on the machines.

@edwardpeek-crown-public
Copy link

We'd like the system config to be considered as well (/etc/xdg/pip/pip.conf on Ubuntu). Our organisation configure a local mirror for all dev machines & users this way.

@RichardDally
Copy link

The environment variables are nice, but reading the pip config file would be very good for enterprises. It would allow a seamless transition to uv without installing additional configuration on the machines.

Exactly, this is what I was expecting from uv, a drop-in replacement with a major performance boost.

@zanieb
Copy link
Member

zanieb commented Mar 5, 2024

This is still under consideration, but the comment at #1789 (comment) explains our current stance well.

@RichardDally
Copy link

Hi @mcrumiller you can use UV_INDEX_URL instead. We may want to add all of our environment variables to the README, but they are currently listed in the --help output e.g.


  -i, --index-url <INDEX_URL>

          The URL of the Python package index (by default: https://pypi.org/simple)

          

          [env: UV_INDEX_URL=]

I've been trying environment variables but it raises an error related to Unexpected fragment on URL: contact.
I've seen issues that have been closed for this, I'm a little bit puzzled here.
Using version 0.1.12 on Windows on private Artifactory repository.

@zanieb
Copy link
Member

zanieb commented Mar 22, 2024

@RichardDally could you please create a new issue if this is not related to pip.conf support?

@zanieb
Copy link
Member

zanieb commented Mar 22, 2024

Okay, we are officially considering support for this feature. Here is my proposal.

Require explicit selection of configuration files

pip supports discovery of configuration files at various locations

  • Do not support automated discovery of pip.conf
  • Do not support global / user / site
  • Use of pip.conf requires opt-in e.g. --pip-config <path> or UV_PIP_CONFIG=<path>

Only support index configuration

pip supports all of the command-line options via the configuration file

  • Do not support flags like no-compile, etc.
    • Unknown flags are ignored without warning
    • Flags we support in the CLI but not in the config file result in a warning in verbose mode
  • Support for:
    • find-links, index-url, extra-index-url
  • Possible support for native-tls toggle

Do not support configuration per command

pip supports separate configuration for each sub-command like install

  • All configuration must be either in a [global] declaration or in the top-level
  • Declarations in the top-level will override those in a [global] section
    • We could alternatively error or avoid checking [global] at all if a top-level option exists
  • Configuration per command will be silently ignored
  • If configuration file is provided without [global] or top-level options we will warn it is empty

@kkpattern
Copy link

Require explicit selection of configuration files

pip supports discovery of configuration files at various locations

  • Do not support automated discovery of pip.conf
  • Do not support global / user / site
  • Use of pip.conf requires opt-in e.g. --pip-config <path> or UV_PIP_CONFIG=<path>

Maybe UV can auto discovery pip.conf in a virtual environment?

UV_INDEX_URL is actually enough for most use cases. I think many want uv to support pip.conf because they want uv as a drop-in replacement for pip. Without auto-discovery pip.conf from a virtual env, this cannot be achieved.

Also, in our project's CI job, there're multiple stage install different dependencies into the same .venv. For example, the bootstrap script may setup the .venv and install all the essential dependencies. Then in a later stage, the test script will install all the test related dependencies into the .venv.

By auto-picking up the pip.conf in the virtual environment, all the jobs in the following stages don't need to worry about which index it should use, which is very convenient for us.

@zanieb
Copy link
Member

zanieb commented Mar 29, 2024

Maybe UV can auto discovery pip.conf in a virtual environment?

What exactly do this mean? There's a pip.conf file in the .venv folder?

@kkpattern
Copy link

What exactly do this mean? There's a pip.conf file in the .venv folder?

Yes. If you activate a .venv then run pip config --site set install.index-url "http://pypi.internal.com", there will be a pip.conf in the .venv containing following content:

[install]
index-url = http://pypi.internal.com

Later pip commands will automatically pick up this config file.

@zanieb
Copy link
Member

zanieb commented Mar 29, 2024

Why not just export an environment variable selecting the configuration file? e.g. UV_PIP_CONFIG=./.venv/pip.conf?

@kkpattern
Copy link

It's convenient when we want to install some libs into a .venv across multiple sessions.

For example, a developer may set up a local .venv for development and use it for a relatively long time. Then, one day, they may need to update or install a new lib from an internal Pypi index. Since pip can automatically pick up the pip.conf file in the .venv folder, they don't need to remember what the index is or where the pip.conf is located.

This is not a deal breaker. We already use uv in our project and are very happy with it. But I think it's a nice-to-have feature.

@zanieb
Copy link
Member

zanieb commented Mar 29, 2024

Thanks for the additional details! I don't think we'd launch this feature with support for that, but I'm not fully opposed. I worry about sometimes supporting inference, but having it work in the most-scoped context makes some sense.

@msharp9
Copy link
Author

msharp9 commented Apr 3, 2024

Since I first created the issue there's been a lot of conversation, this is what I would suggest as I've thought about everyone's different viewpoints.

UV should create it's own configuration file and provide automatic discovery for it.
There should then be another tool that converts pip configs to uv configs, e.g pip.conf -> uv.conf.
This second tool could provide a simple extensible configuration to add default locations. Collect all pip.conf, and create one combined uv.conf to the auto discovery location.

If UV then added support for plugins, then all a user would have to do is pip install both, say, pip install uv uv-pip-conf. This would make the user experience feel much more like UV is actually a drop in replacement to pip and pip-tools since no flags or environment variables would have to be set.

This would be similar to how flake8 plugins work, simply pip install flake8 and flake8-import-order as an example, and now flake8 automatically checks your codes imports along with other syntax.

@d-miketa
Copy link

d-miketa commented Apr 3, 2024

@zanieb Slightly off topic, but would you be willing to share an expected timeline for the config file feature? It’s the one thing preventing me from immediately swapping all of my projects to uv. 😅 Many thanks, love the project.

@zanieb
Copy link
Member

zanieb commented Apr 3, 2024

Hey @d-miketa — we don't have a timeline yet. I think it will be relatively straight-forward to implement but I'm giving time for feedback on my proposal which only has four upvotes compared to the ~50 on the original post.

@gaby
Copy link

gaby commented Apr 14, 2024

As stated by @PetterS not having built-in support for /etc/pip.conf will be detrimental for Enterprise adoption.

In our case we drop a configuration in /etc/pip.conf using Ansible, we then symlink this file to /etc/xdg/pip/pip.conf for Ubuntu based systems.

After that any Python package installed using ansible.pip module automatically uses the configuration from /etc/pip.conf, we don't have to expose ENV's or write extra config files.

@charliermarsh
Copy link
Member

That requirement sounds more like you need a persistent configuration file rather than a pip.conf specifically - or am I misunderstanding?

@gaby
Copy link

gaby commented Apr 14, 2024

@charliermarsh In enterprise environments /etc/pip.conf is used to managed several things:

  • Index-url
  • Search-index
  • HTTP/HTTPS proxies
  • Custom CA's

User are just able to do "pip install" without having to configure anything since it's managed by the platform via /etc/pip.conf.

@astronautas
Copy link

astronautas commented Apr 18, 2024

Any progress on this? It's critical for enterprise environments.

@CostcoFanboy
Copy link

CostcoFanboy commented Apr 20, 2024

The intent is to introduce our own configuration in the future and we'll support automatic discovery of that. We're just not particularly comfortable reading other tool's configuration files without opt-in

Then uv will stay indefinitely a non-viable product for enterprise solutions.

/etc/pip.conf or even sometimes pip.conf files getting cp'd inside of the venv folder during the dependency installation phase is an industry-wide practice/trend, for better or for worse.

Regardless, huge fan of the product and thank you immensely for your time and hard work.
Just sad to see this will only stay on personal projects.

@charliermarsh
Copy link
Member

I think what I don't understand yet is: if you're going through the effort to migrate from pip to uv, why is it any harder to ask that you use (for sake of argument) uv.conf rather than pip.conf? Why is that not possible? Like, if I told you that it was the exact same file, but it had to be named uv.conf, why would that be a problem?

We may be convinced to change the behavior here (I'm genuinely looking for answers to that question), but FWIW I can personally name a bunch of enterprises that are already using uv, so I'm not worried about it only being viable for personal projects :)

@gaby
Copy link

gaby commented Apr 20, 2024

@charliermarsh If uv.conf becomes a thing, and location is predefined, ex /etc/uv.conf. It should work enterprise environments.

@charliermarsh
Copy link
Member

That’s the thing I’m mostly trying to understand. Is it important the file is literally named pip.conf, that it follows the exact same schema, that it’s auto-discovered in the same way? If so, why? (I’m genuinely open to being convinced, it just adds work and a maintenance burden for us so I want to make sure we have the right motivation.)

Or is it that users just need a way to specify persistent configuration? What if we (eg) supported reading pip.conf but you had to set a UV environment variable for it?

@CostcoFanboy
Copy link

I think what I don't understand yet is: if you're going through the effort to migrate from pip to uv, why is it any harder to ask that you use (for sake of argument) uv.conf rather than pip.conf? Why is that not possible? Like, if I told you that it was the exact same file, but it had to be named uv.conf, why would that be a problem?

Apologies I don't think I expressed myself clearly.

While the actual name of the file doesn't necessarily need to be the same, the feature set need to be mirrored and it would be highly desirable for the discovery mechanism to be mimicked. Mostly having a global predefined location (e.g. /etc/pip.conf) and a project-specific/localized one (e.g. inside of venv in its root) is very important..

From the previous posts it didn't feel like there was functionality-mirroring in mind, but from your clarification it does feel like I'm wrong.

FWIW I can personally name a bunch of enterprises that are already using uv, so I'm not worried about it only being viable for personal projects :)

Would this mean there is another way that UV supports trusted-host, index-url AND extra-index-url (fallback)?

@hutch3232
Copy link

In my enterprise environment, sys admins configure /etc/pip.conf to work with our PyPI mirror. A sizeable portion of our users probably do not really even understand the mirror or bother with pip.conf since it "just works". Thinking short-term, it would be nice if uv picked up pip.conf so that it was a drop-in replacement for pip in enterprise environments. It wouldn't require convincing the sys admins to create/maintain an additional config file or to teach users how to do it themselves. Longer-term though, as uv adoption increases, it probably wouldn't be much of an ask for sys admins to create the file. I think supporting pip.conf would primarily help less comfortable users get up and running with uv. More advanced users would have little issue figuring it out, using a hypothetical uv.conf or environmental variables. With that in mind, less comfortable users probably are sticking with pip in the short-term anyway. More advanced users are probably the people very excited about uv and jumping into it.

@astronautas
Copy link

@charliermarsh In our case (a small data point, but still) we don't care that much how the file is named, as long as uv can resolve deps from non pypi sources and it's possible to define those sources within *.conf.

@PetterS
Copy link

PetterS commented Apr 21, 2024

There can be (and have been where I've worked) automated setup scripts that make sure there is a pip.conf with appropriate settings. If uv just works with that file the gradual adoption within the enterprise could be quicker.

It's not a dealbreaker of course since we are talking about developers here.

@eswolinsky3241
Copy link

I think what I don't understand yet is: if you're going through the effort to migrate from pip to uv, why is it any harder to ask that you use (for sake of argument) uv.conf rather than pip.conf? Why is that not possible? Like, if I told you that it was the exact same file, but it had to be named uv.conf, why would that be a problem?

We may be convinced to change the behavior here (I'm genuinely looking for answers to that question), but FWIW I can personally name a bunch of enterprises that are already using uv, so I'm not worried about it only being viable for personal projects :)

@charliermarsh Just to comment since I came across this post while playing with uv: We also use AWS Codeartifact, which uses short-lived credentials that need to be refreshed every twelve hours. The AWS CLI offers a single helper command that can obtain the credentials and rewrite the index url in pip.conf (or the equivalent config file for npm, maven, etc.). Obviously AWS currently offers no such support for uv, so it's on the user to update uv.conf every twelve hours. Not saying this is a huge deal, just something that could be tedious enough to convince an enterprise user to stick with pip. Loving uv so far though!

IceCodeNew added a commit to IceCodeNew/docker-collections that referenced this issue May 26, 2024
@ilCatania
Copy link

I also work in a corporate environment where someone else manages pip.conf on my behalf and I do not have full control over my users' installations.

My short term plan to be able to replace pip with uv pip has been a handmade utility that reads from pip.conf and sets the corresponding uv environment variables.

I would be very happy, once uv supports its own configuration file, if there was a command to create / update said configuration file with values from pip.conf. Something like

uv import-pip-conf --overwrite --ignore-unsupported --pip-conf-file=~/.pip.conf

(I am making it up but you get the idea hopefully)

@quartox
Copy link

quartox commented Oct 18, 2024

I also have the short-lived credentials problem in an corporate environment. My pip.conf file is updated every 8 hours with new credentials. The tools to update these credentials are owned by other teams at my company. It would be more challenging for me to convince them to adopt uv before I am able to test it out and demonstrate some improvements with existing workflows.

I was wondering if it would it be possible for uv.conf to reference specific configurations in pip.conf? For example what if the uv.conf file contains the line

index-url = ${/path/to/pip.conf:index-url}

Don't care much about specific syntax (which I took from here) but the intent is to reference values in the pip.conf file explicitly. This way we could keep updating pip.conf every 8 hours until I can get broad consensus to move to uv.conf workflows.

@T-256
Copy link
Contributor

T-256 commented Oct 22, 2024

FYI, #7851 landed, which now allows you to set global configuration for uv.toml.

@darksim33
Copy link

FYI, #7851 landed, which now allows you to set global configuration for uv.toml.

Just to make this clear since I'm new to the tool. Is it possible to link the pip.conf using the uv.toml?
I tried a few things but they don't seem to work.

@gaby
Copy link

gaby commented Oct 25, 2024

@charliermarsh In enterprise environments /etc/pip.conf is used to managed several things:

  • Index-url
  • Search-index
  • HTTP/HTTPS proxies
  • Custom CA's

User are just able to do "pip install" without having to configure anything since it's managed by the platform via /etc/pip.conf.

Quoting myself since I just tried using /etc/uv/uv.toml and some of these options are not even supported.

uv keeps pushing some of these configs to ENV variables. These are not practical for enteprise environments.

Currently there's no way to set a cert file for a proxy/index, and there's no way to set http/https proxies via pyproject.toml / uv.toml

Enteprise environments typically set SquidProxy or Pypi Mirror. These services are exposed to users with internal TLS certs signed by the Enterprise CA.

tldr: If the ENV's listed here https://docs.astral.sh/uv/configuration/environment/ were supported via uv.toml and pyproject.toml it would solve a lot of the use-cases listed on this thread.

@schlamar
Copy link

@gaby for this use case you can use native-tls setting https://docs.astral.sh/uv/reference/settings/#native-tls

In this case SSL works automatically if your internal cert is in the system's certificate store (which is usually the case in enterprise setups).

@gaby
Copy link

gaby commented Oct 25, 2024

@schlamar Yes, that works. But sometimes the Proxy CA is not installed in the system.

Regardless, you can't set http/https proxy via toml only via ENV.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compatibility Compatibility with a specification or another tool configuration Settings and such
Projects
None yet
Development

No branches or pull requests