Implement `--user` flag and user scheme support for `uv pip` #2352

imfing · 2024-03-11T08:06:26Z

Summary

Hi team, excited for your awesome work of uv.

This PR attempts to implement --user flag for uv pip subcommands: install, freeze, list and show.

Closes #2077

Per PEP 370 – Per user site-packages directory, pip supports user scheme package installation which would be useful for multi-user environment or CI use cases without the need of virtual environment or breaking system Python packages.

Example usage:

export PYTHONUSERBASE=~/.local/myappenv
uv pip install --user SomePackage
# SomePackage is available under ~/.local/myappenv

Implementation details

The implementation aims to bring minimal changes to the existing functionalities of uv pip, as well as supporting --user for pip compatibility.

Here are some breakdowns of the implementation:

add _infer_user() to crates/uv-interpreter/src/get_interpreter_info.py which returns the user scheme if specified
add PythonEnvironment::from_user_scheme() which queries the interpreter info with user scheme, thus the platlib, purelib, scripts, data and include should point to the user site
use the user scheme Python environment for uv pip subcommands when the --user flag is present, making the subcommands use only the user site for package search and installation.

Right now the implementation doesn't include the global packages mentioned in User Installs since we only consider site packages from the scheme.purelib. I'm thinking about adding that later for better compatibility with pip install.

I am new to uv, looking forward to the team's feedback and suggestions :)

Test Plan

May need some advice and guidance on what's the best practice on testing this.

* to use requested python, virtualenv and default python in order * ensure the scheme directories exist

crates/uv-interpreter/src/python_environment.rs

Co-authored-by: konsti <konstin@mailbox.org>

charliermarsh · 2024-03-11T17:54:17Z

This looks like a nice PR! The main blocker is that I need to decide if we want to support --user. I think it leads to some limitations in pip but I just haven't had time to investigate it. For example, it was suggested elsewhere that --user and --editable are no longer supported in pip -- is that true?

charliermarsh · 2024-03-11T21:20:30Z

crates/uv-interpreter/src/interpreter.rs

+    pub(crate) fn query_user_scheme_info(interpreter: &Path) -> Result<Self, Error> {
+        let script = include_str!("get_interpreter_info.py");
+        let envs = vec![("_UV_USE_USER_SCHEME", "1")];
+        let output = Self::execute_script(interpreter, script, Some(envs))?;


Does this intentionally not cache?

I've noticed that query_cached uses the executable's last modified time as a cache key. In the case of user scheme, the interpreter may remain the same, e.g. the system Python interpreter, but the InterpreterInfo.scheme changes to the user site, e.g. ~/.local.

I will take a further look into using uv_cache to cache the script execution here.

charliermarsh · 2024-03-11T21:21:13Z

Again, just want to say: this is very nice work! In fact I'm more inclined to support this now that we have a nice PR for it. But I do want to understand the implications a little more first.

charliermarsh · 2024-03-11T21:21:39Z

crates/uv/src/commands/pip_install.rs

+    // <https://github.com/pypa/pip/blob/a33caa26f2bf525eafb4ec004f9c1bd18d238a31/src/pip/_internal/commands/install.py#L686>
+    if user
+        && venv.interpreter().is_virtualenv()
+        && !venv.cfg().unwrap().include_system_site_packages()


Can we use is_some_and or unwrap_or here instead of unwrapping?

imfing · 2024-03-12T07:43:23Z

Thank you for the comments 😄

For example, it was suggested elsewhere that --user and --editable are no longer supported in pip -- is that true?

It seems there might be some confusion stemming from an issue discussed here: #2077 (comment)

One of the problems is that you cannot run --user and --editable any more with pip.

As far as i know, both --user and --editable are still supported in the latest pip install options.
There appears to be no intention from the pip developers to deprecate these flags. I couldn't find any discussion about this either.

I don't have the full context so the complexity might arise specifically in the context of using Airflow. It would be great if the author of that comment could provide some further details.

For reference, I conducted a quick local test combining the --user and --editable flags:

❯ python3 -m venv venv --system-site-packages
❯ source venv/bin/activate
❯ pip install --user -e sampleproject/
Obtaining
  Installing build dependencies ... 
...
Successfully built sampleproject
Installing collected packages: peppercorn, sampleproject
...
Successfully installed peppercorn-0.6 sampleproject-3.0.0

❯ pip list --user
Package       Version Editable project location
------------- ------- -----------------------------------------
peppercorn    0.6
sampleproject 3.0.0   /Users/me/sampleproject

zanieb · 2024-05-17T13:46:56Z

There's some more discussion on this active in #2077

charliermarsh · 2024-05-17T14:16:30Z

I may expand the section on --user in PIP_COMPATIBILITY.md in the interim.

See: #2352

Malix-Labs · 2024-06-05T12:04:37Z

Is there any blocker that requires some help to merge this PR?
We are waiting for that feature to be merged to migrate every devcontainer we have to use uv

imfing · 2024-06-05T13:18:16Z

Is there any blocker that requires some help to merge this PR? We are waiting for that feature to be merged to migrate every devcontainer we have to use uv

In #2077, there were extensive discussions, and from what I gathered, it seemed the uv team prefer advocating for the adoption of virtual environments in uv.

I kept this PR open in case things change, and if the uv team is okay with adding the flag, I will update the PR.

zanieb · 2024-06-05T13:44:35Z

@Malix-off can you please discuss this on the issue instead of this pull request? I'd prefer to keep the discussion here focused on implementation details. Thanks.

imfing · 2024-06-05T14:00:34Z

Noticed that --target flag was added in #3257, which can be used as a workaround for --user by providing the user-specific site-packages directory

imfing added 20 commits March 9, 2024 16:29

Initial implementation of user scheme

a6a4b77

Ensure venv scheme paths exist

9c35a6a

Merge remote-tracking branch 'upstream/main' into user-scheme

62ba8c9

Fix merge issues

425f4f7

Add include_system_site_packages to PyVenvConfiguration

5cef5d8

Refactor to from_user_scheme

bbd910d

* to use requested python, virtualenv and default python in order * ensure the scheme directories exist

Update pip install --user arg docstring

642c987

Merge remote-tracking branch 'upstream/main' into user-scheme

8cf2d97

Support user site in pip list

dcc7c29

Support user site for pip show

0bb5b6a

Support user site for pip freeze

7bc1895

Fix python arg type conversion

ad97a62

Format file

947f93b

Merge remote-tracking branch 'upstream/main' into pip-user-scheme

dea359c

Update comment for pip install user arg

2f905e4

Add comments to _infer_user

880da62

Add include dir to ensure user scheme paths

d2df185

Merge remote-tracking branch 'upstream/main' into pip-user-scheme

7a9af2f

Run cargo fmt --all

eb38890

Fix CI errors

97bf336

konstin added the compatibility Compatibility with a specification or another tool label Mar 11, 2024

konstin requested a review from charliermarsh March 11, 2024 08:49

konstin reviewed Mar 11, 2024

View reviewed changes

crates/uv-interpreter/src/python_environment.rs Show resolved Hide resolved

imfing and others added 2 commits March 11, 2024 09:03

Update crates/uv-interpreter/src/python_environment.rs

1991f70

Co-authored-by: konsti <konstin@mailbox.org>

Use fs_err instead of fs

6749fa7

charliermarsh reviewed Mar 11, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into pip-user-scheme

67f4821

imfing mentioned this pull request Mar 12, 2024

Support --user flag from pip #2077

Open

imfing added 2 commits March 13, 2024 23:57

Merge remote-tracking branch 'upstream/main' into pip-user-scheme

f6e9a43

Fix merge issues

5f49ffc

charliermarsh mentioned this pull request May 20, 2024

Explicitly mention --user in compatibility guide #3666

Merged

charliermarsh added a commit that referenced this pull request May 20, 2024

Explicitly mention --user in compatibility guide (#3666)

d7dc184

See: #2352

This comment was marked as off-topic.

Sign in to view

This comment was marked as resolved.

Sign in to view

imfing closed this Jun 15, 2024

imfing deleted the pip-user-scheme branch June 15, 2024 11:43

zanieb mentioned this pull request Jun 28, 2024

Bad experience: Mac, python from brew, pip install uv #4624

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `--user` flag and user scheme support for `uv pip` #2352

Implement `--user` flag and user scheme support for `uv pip` #2352

imfing commented Mar 11, 2024 •

edited

Loading

charliermarsh commented Mar 11, 2024

charliermarsh Mar 11, 2024

imfing Mar 12, 2024

charliermarsh commented Mar 11, 2024

charliermarsh Mar 11, 2024

imfing commented Mar 12, 2024

zanieb commented May 17, 2024

charliermarsh commented May 17, 2024

Malix-Labs commented Jun 5, 2024

imfing commented Jun 5, 2024

This comment was marked as off-topic.

zanieb commented Jun 5, 2024

This comment was marked as resolved.

imfing commented Jun 5, 2024

Implement --user flag and user scheme support for uv pip #2352

Implement --user flag and user scheme support for uv pip #2352

Conversation

imfing commented Mar 11, 2024 • edited Loading

Summary

Implementation details

Test Plan

charliermarsh commented Mar 11, 2024

charliermarsh Mar 11, 2024

Choose a reason for hiding this comment

imfing Mar 12, 2024

Choose a reason for hiding this comment

charliermarsh commented Mar 11, 2024

charliermarsh Mar 11, 2024

Choose a reason for hiding this comment

imfing commented Mar 12, 2024

zanieb commented May 17, 2024

charliermarsh commented May 17, 2024

Malix-Labs commented Jun 5, 2024

imfing commented Jun 5, 2024

This comment was marked as off-topic.

zanieb commented Jun 5, 2024

This comment was marked as resolved.

imfing commented Jun 5, 2024

Implement `--user` flag and user scheme support for `uv pip` #2352

Implement `--user` flag and user scheme support for `uv pip` #2352

imfing commented Mar 11, 2024 •

edited

Loading