-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support --user
flag from pip
#2077
Comments
Thanks! I really appreciate the clear write-up and motivation here. (I'm hoping to get to this today, it's been requested a few times.) |
Just to take off the pressure a bit. I figured out how to get rid of the So ... It seems I do not need it that much any more .... I will make Airflow PROD image also There is a small difference though with venv when using After that experience I have a new thought. The |
@potiuk - At the very least I can add |
Yep. That would be a good start :). I think |
Yeah I strongly agree with this. Very good callout. It's nice to add things that unlock user workflows, but there are some areas where we actively want to change user behavior, and we need to hold firm on some of those lines. |
Can I ask why you use |
Yes. In Airflow we have something called It creates a new virtualenv based on requirements, python version. One of the options (only valid if your python version matches the This is super important, because we serialize arguments that we pass to the operator, and de-serialze return value from it, as interface between airflow and the "new venv" method executed. And in order to serialize some objects, the target venv should have the "airflow + often other packages" installed. Moreover the python code to be executed (basically a methpd) can do some local imports - expecting airflow or other packages to be installed. The Here howto: https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator |
Under the hood, the operator does:
|
The way how it works now (just built and tested a new image): With system-site packages:
Without:
|
@potiuk -- I'm trying to make a decision on whether to support |
The starting point I have is this: And I think there were lots of related discussions: Here is one: pypa/pip#6375 Looong discussions. But basically the gist of it is that Personally I consider But I am not fully aware about some history behind We luckily got rid the |
@potiuk Thank you for this issue, and thanks for the inputs you provided Some of my thoughts regarding the comment above:
using the
I skimmed through it, and it looks like the issue was mainly about From what I understand, PEP 517 itself discusses allowing projects to specify different build systems through
If we do a quick search on GitHub for the Though I haven't used Putting PR #2352 was my preliminary attempt to fit it into the I haven't looked into every combination of how Ultimately, it's up to the uv team to decide whether to support it. |
This is an excellent quote, thanks @imfing for noting this. As somebody coming from the scientific Python world, I was meaning to write something along these lines in this thread as well. Another useful framing of this is the distinction between a python project versus a single-file script. For the latter, having a separate venv for each individual script would not only be annoying but also error prone, as you'd need to be constantly activating / deactivating environments, when often all you need is stdlib and numpy/scipy. We also have a rather odball use case for this -- we have a very unholy setup where we provide a system python installation in a container image with a lot of pre-installed packages, but we allow users to install additional packages to Small side note: If this feature is accepted, I think providing (all that said, it's brilliant that uv defaults to requiring a venv, that's already a huge change with respect to pip, and forced me personally to start using them, where previously I've used conda + pip, not a great combo. I never quite got myself to understand the various subtleties - venv x virtualenv etc.) |
Note this perspective is something that we could solve with new workflows rather than matching the existing It's worth keeping in mind throughout this discussion that we will be building new workflows on top of the fundamental tooling we've built for pip-compatibility. If we can avoid implementing and maintaining something with complex compatibility concerns we can focus on innovation and comprehensive solutions. |
This is also my concern - I think (and Airflow use-case proves that) implementing something like that and having users rely on it should be conscious decision, because it will stay with And I re-read that article again (it's really funny to read such an article few years later and confront your current views with the views of few-years-younger yourself). Suprisingly (or maybe now), I tend to agree with that I think PEP-370 proposal was good, and the reason described above (some people do not care about venv and they want to do things quickly) is very valid (recognising that non-power users want to just get-on with their install is also an argument in my article actually). But the discovery I made in apache/airflow#37785 that I can simply create a venv in What happens currently when you start Is it good for non power users ? Not at all. It's confusing for first time users who have no knowledge about But wait - we already have a way to create venvs (fast) in That would be my proposal in short:
|
Just to say - I too would be very interested in a mechanism to support beginners who do not yet know about virtualenvs. I have taught a lot of students to use Python for data analysis, including data science. In practice these beginners (and there are lot of them) will not need to (consciously) make virtualenvs for a while - because the standard Python data stack is pretty stable, and doesn't generate many dependency conflicts. I'd estimate these beginners can make six months or more of progress before they need to start thinking about making and switching virtualenvs. I think it will be a significant barrier if they have to learn virtualenvs and their structure before they install their first Python package. So I would love a solution like @potiuk's - where the user can ask for, or even just be given, a default virtualenv, into which they install their packages. Of course the |
In case it's helpful - the use-case I have in mind is - I believe - very common - and that is the beginner starting a class or an online tutorial. For example, consider this tutorial: https://realpython.com/pygame-a-primer/ It has the instructions, right at the top, of:
Or - for my students, I suggest:
In both cases, this gets the beginner going with early Python work. But - with various installation methods - including
Of course this isn't going to worry an experienced user, but it will be confusing to a beginner, who does not know what a virtualenv is - and - for a beginner, who doesn't understand Python installation structure, the virtualenv is a relatively advanced subject. So - it would be very good to have some default set up, such that the beginner would not face this learning curve immediately, but face it later, when their needs and experience have expanded. I'm posting here because I have, up until now - suggested my students do e.g. |
Maybe the topic, of whether or not to add a New users, as @matthew-brett mentions, are often given "pip install x" instructions. This should ideally just work, and definitely not raise obscure errors that confuse new users who know nothing about virtual envs. Ideally, I'd imagine something like: if in venv, install there, otherwise install into a local default venv. The local default venv has access to system packages, but shadows them, so that the user can upgrade packages. This would make for a simple first user experience, compatible with all tutorials out there, without impacting sophisticated users who can set up their own venvs. |
Yep. My proposal is that it will just install things in |
I would definitely love that When you're already working in a containerized development environment (nix shell / devcontainer …), it makes perfect sense (and even should be the default). |
I wonder though, whether anyone would worry about previous |
Do that mean that the python ecosystem has some dependency on venv usage ? |
More that: |
|
No that doesn't help. Please see the STR in unbit/uwsgi#2525 (we're getting a bit off-topic here, so this will be my last reply :-) ) |
I was (as you probably saw) following the discussion over at Scipy - could you say more about the current state of play for this? Is there a recommended |
The current state is that you need declarative metadata, either in a project (via |
Do you think there will be a route to using |
Frankly, i don't understand how this conversation went on discussing if this feature should be implemented at all when no non-convoluted alternative like imfing's suggestion is given. To illustrate my point; here's my first experience playing with uv:
And i'm already looking at obscure Github discussions to make a common Python tool available outside of a development environment. |
You might be starting from the wrong conclusion? You either want a virtual environment (as in the hint!) or to install your package via |
Were you thinking of any particular part of that page as an answer to the question? For context - I don't think the question was - how do I install a command line tool in an isolated environment? - it was how do install a set of particular packages into any kind of default environment - such as the system environment, or a user environment. |
For context - I strongly suspect that there are always going to be a significant group of people for whom "install in a virtualenv" is not going to be a good answer - and very few of those people are trying to install an isolated command line tool. My guess is that, given the move of both |
Rotonen said: charliermarsh said:
First off, thanks for pointing me in the right direction. After all "tools" worked exactly as I needed, which is great, but I’m still a bit confused about a couple of things:
|
@63427083 -> I heartily recommend you listen to the podcast of @charliermarsh https://realpython.com/podcasts/rpp/238/ -> he eplains there the reasoning and the whole philosophy behind uv. Uv is not what
I don't think the goal of uv is to allow you to install packages in a standalone, separate .venv installed in a "default" python. This is what used to be the "old" way how you used to work with Python, one that has been very problematic and everyone (including pip 24+) will not let you follow that old pattern without specifying some specific flags. Basically all the tools out there introduce "friction" so that people re-learn how to deal with python installation differently. Even if it means that those people will be unhappy and have to change their habits. This is - as I understand it - very deliberate decision, that follows (or even sometimes leads) the way Python Packaging is evolving, and to be frank, you will be better to change your habits to follow those patterns. And yes, I understand you might be unhappy, but I strongly recommend you to listen to the podcast, it provides much more context and explanation and after listening to it, you might be more willing to adapt. |
Speaking of which @charliermarsh -> there is something to the messaging question that @63427083 had -> maybe it's worth to provide more guidance to users who also have expectations about "working the same way as old ways". |
Yes, no question - that is the move - and the move is, I believe, based on two assumptions:
I can see that these assumptions may be tempting to an experienced developer, but I can tell you, having taught many many beginners, that neither of these assumptions is correct. And so, for beginners, we'll need something else. I'm happy to explain why the assumptions are not correct for beginners, but I don't want to extend this thread for those who aren't working with beginners. |
A note: it’s not just a flag from pip, it’s the concept of user packages specified in PEP 370 and compatible with multiple tools. One example: on debian and derivatives, bash includes ~/.local/bin in PATH by default if it exists. So pip install --user sometool is a valid and easy way to be able to run sometool without more docs or setup. |
But the thing is that Also if you want to develop something with python you do:
As a beginner, you don't even have to have python installed, Again - it can't get any simpler than that. So maybe your comments apply to the ways how beginners used to have problems with starting with Python before
And yes, I am (painfully) aware of PEP 370 and I am aware how universally problematic it turned out to be for |
And again (@matthew-brett ) - go and listen to the podcast. Charlie explains exactly why |
By "beginner" I mean someone who is expecting to start an interactive Python console of some sort and do something like
Well - yes it can. Because a) that won't match nearly all tutorials out there, and b) it depends on the user keeping in mind that they need to stay in the same directory (or subdirectory thereof) to get the relevant virtualenv. So if they, for example, do Just for reference, in a course we taught in Berkeley, to stats students, we had to include a class on the filesystem in the terminal, because many inexperienced computer users have a hazy idea of how files are arranged on the computer, beyond the idea of a "Folder". I find that many developers haven't come across these kind of problems, and indeed they are easy to miss if you don't have quite close contact with early learners. |
I think this exchange is sufficient — you've communicated your opinions on this topic and I'll ask that we don't debate these specific points more here. We care a lot of the beginner user experience and we have several team members with scientific backgrounds where "beginner" level Python experience is quite common. However, we're adverse to repeating the mistakes of the past and are interested in pursuing new norms to make workflows easier. I'll admit that it's not perfect today, there's more to do still. I've created two issues to track actionable improvements to @63427083's comment: |
I very much concur wiht @zanieb 's comment and the follow-up. And seeing those, as the author of the issue I consider it "closed", so I am going to do exactly that. |
Currently (even in latest uv 0.1.12 that supports --python flag) it's not possible to use
uv
in case--user
flag ofpip
(or PIP_USER="true" env variable is set).With
--user
flag, packages are installed to a ~/.local folder, which means that the user might not have access to thepython
system installation as a whole, but can locally install and uninstall packages, without having venv.This is extremely useful for cases like
CI
and building container images, where having a venv is an extra overhead and it is unnecessary burden.There is an interesting (and used in Apache Airflow) case for the
--user
flag (this currently prevent us from usinguv
by us and our users also in PROD images in addition to our CI image). That is currently a blocker for apache/airflow#37785Why
--user
flag is useful and why we use it in Airflow?Using
--user
flag is pretty useful when you want to have an optimized image - because such.local
folders can be copied between different stage of the image (based on the same base image and prod libraries installed) and you can copy the whole.local
folder between the stages after packages are installed - which means that the final stage of the image does not have to havebuild-essentials
/ compilers installed.You could do the same with
venv
if you keep it in the same folder, but that looses an essential capability of creating venv dynamically containing all the packages you have in your.local
folder. When you have your--user
installed packages, and you create a new venv with--system-site-packages
, the packages installed in.local
folder are also installed in the new venv. This does not work when you create a newvenv
from anothervenv
because--system-site-packages
are only the ones installed in the system.While I can think of some creative ways (maybe I will find some) - having an equivalent of
--user
installation byuv pip install
would be a great simplification for our case to support the users who want to useuv
(and seems that there is already a need for that looking at the apache/airflow#37785The text was updated successfully, but these errors were encountered: