Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an option to store virtual environments in a centralized location outside projects #1495

Open
DanCardin opened this issue Feb 16, 2024 · 51 comments
Labels
enhancement New feature or improvement to existing functionality projects Related to project management capabilities virtualenv Related to virtual environments

Comments

@DanCardin
Copy link

I ideally dont .venv/ folders cluttering my project folders (not the least because they may not be safe to move, at least with venv)

In my personal (also rust) workflow tool (that i'd love to not have to maintain if uv could arrive at some of the same decisions 😆) there is a setting which when set to "central" puts all venvs in $XDG_DATA_HOME/<toolname>/..., and the path to the venv is determined by mirroring the path to the project. That is, ~/foo/bar/baz/pyproject.toml -> $XDG_DATA_HOME/uv/foo/bar/baz/.venv/.

By comparison to more traditional tools, poetry also (by default) will automatically create venvs in a central location. Although it puts all venvs in the same folder using a somewhat inscrutable hash to disambiguate projects of the same name.


This ^ feature sort of implies a few other mostly separate features in order to be useful:

  • uv activate (because it becomes impractical to self-activate, except by copy-pasting the path that gets printed)
  • which then implies some uv self init --shell zsh (or whatever), that gives you the shell integration to automatically activate through the CLI itself.
  • uv venv --delete or -d
@zanieb
Copy link
Member

zanieb commented Feb 16, 2024

Thanks for the issue!

This is the kind of thing we plan to tackle in the future, i.e. when we build an opinionated workflow for environment management. It's not in scope right this second, but we'll revisit it :)

@zanieb zanieb added enhancement New feature or improvement to existing functionality projects Related to project management capabilities labels Feb 16, 2024
@woutervh
Copy link

woutervh commented Feb 16, 2024

By comparison to more traditional tools, poetry also (by default) will automatically create venvs in a central location. Although it puts all venvs in the same folder using a somewhat inscrutable hash to disambiguate projects of the same name.

IMHO:
It's one of the really bad decisions of poetry.
As a python contractor usually supporting other python developers, I really dislike any situation where devs have no idea where their virtualenv of their project is. Your executables are the most important thing in your project, hiding them in arcane directories should never be a default.

Activating a venv is an antipattern because it introduces state in your shell.
I really hope it can be de-emphasized in the future.

@DanCardin
Copy link
Author

Imo there are various positive sideeffects of the venv not being local, although I dont personally like their chosen algorithm for selecting it.

But ultimately, as long as the tool can know where the venv should be given the settings/invocation options, then there isn't generally a need to activate the venv, at least for uv itself to function. But it's the reality that many tools require VIRTUAL_ENV to be set to function properly, which either means activating the venv or routing all commands through a uv run (which might be nice interactively but isn't ideal for committed files imo).

@ResRipper
Copy link

I use Windows/Linux/Mac every day and synchronise my projects using OneDrive, having .venv in my project folder is such a nightmare, and that's why I started using pipenv. IMO venv should always be centralised as environments and packages are platform dependent, they are not part of the "content" of the project.

I'm using this plugin to manage and switch between different environments and don't have to know their location most of the time, but I do hope there is a standard for that.

@hauntsaninja
Copy link
Contributor

hauntsaninja commented Feb 17, 2024

Note uv currently appears to work if you make .venv file a symlink to your actual venv

If you don't have symlinks on your platform, this patch of uv may work for you by adding support for .venv files that point to the actual venv location: #1578 (comment)

When uv does go in the higher level workflow direction, I'd advocate leaving a .venv symlink or file pointing to the central location. This makes it easy for IDEs and other tools to figure out where the environment is. This .venv file / symlink idea was discussed around the time PEP 704 was a thing and had fairly positive support

@woutervh
Copy link

@hauntsaninja Thanks for the tip.

I was annoyed that uv does not support the most common normal use-case of virtualenv ootb:

> virtualenv foo
> cd foo
> bin/pip install <package1>

To make it work, the symlink indeed works

> uv venv foo
> cd foo
> ln -s .  .env
> uv pip install <package2>

It would be nice if "uv venv" would make the link by default.

And this would be a nice solution for managing venvs centrally outside the project-folder.

@ResRipper and for syncing via onedrive, you can point .venv to one of the os-specific .venv-files

.venv -> .venv-linux
.venv-linux  --> /some/linux/path
.venv-mac  --> /some/mac/path
.venv-windows  --> /some/windows/path

@zanieb zanieb added the virtualenv Related to virtual environments label Feb 18, 2024
@matterhorn103
Copy link

matterhorn103 commented Apr 16, 2024

Just to add my voice that this would be really really nice to have. Different tools seem to choose either one approach or the other, and it would be great if uv would do both as I like and use both in different situations:

For development situations where git is being used, having .venv there in the project's folder is convenient.

For other projects e.g. scientific ones that are in maybe shared or cloud or working folders, it's undesirable behaviour, and then working with conda is much smoother than with Python venvs. That's pretty much the only thing that keeps me using conda for some things (other than the different ecosystem, naturally).

My impression was that symlinks have poor portability, so like @DanCardin I'd prefer something like a simple -c switch to create the venv in an automatically determined central location, and simple automatic activation, e.g.:

~/example/foo >>> uv venv -c
Using Python 3.11.9 interpreter at: /usr/bin/python3
Creating virtualenv at: /home/jdoe/.local/share/uv/example/foo/.venv
Activate with: uv venv activate
~/example/foo >>> uv venv activate
Activating virtualenv at /home/jdoe/.local/share/uv/example/foo/.venv
(foo) ~/example/foo >>> deactivate
~/example/foo >>>

but with the addition that it would be cool to be able to also activate the venv by name from another location, with the search resolved intelligently and some ability to disambiguate; I could imagine that looking like:

~/some/folder >>> uv activate foo
Searching for virtualenvs at /home/jdoe/.local/share/uv/**/foo/.venv
Activating virtualenv at: /home/jdoe/.local/share/uv/example/foo/.venv
(foo) ~/some/folder >>> uv activate bar
Searching for virtualenvs at /home/jdoe/.local/share/uv/**/bar/.venv
error: virtualenv name is ambiguous! The following matches were found:
  1) /home/jdoe/.local/share/uv/Documents/bar/.venv
  2) /home/jdoe/.local/share/uv/project1/bar/.venv
disambiguate venvs with the same name using parents e.g. to activate 1) use:
  uv venv activate Documents/bar
(foo) ~/some/folder >>> uv activate project1/bar
Searching for virtualenvs at /home/jdoe/.local/share/uv/**/project1/bar/.venv
Activating virtualenv at: /home/jdoe/.local/share/uv/project1/bar/.venv
(bar) ~/some/folder >>>

@matterhorn103
Copy link

So in addition to the "not wanting the venv to be stored in a folder that is backed up to the cloud" use case, I found another use case today:

  • Creating compiled binaries of Qt apps written with PySide using pyside6-deploy (a nuitka wrapper) isn't possible if a venv folder is present in the application folder

(In my view this is a short-sighted approach on the tool's part, but it's just another example of why it might be necessary to keep a venv elsewhere.)

zanieb pushed a commit that referenced this issue Aug 26, 2024
For various reasons, I have a preference for out of tree virtual
environments. Things just work if I symlink, but I don't know that this
is guaranteed, so I thought I'd add a test for it. It looks like there's
another code path that matters (`FoundInterpreter::discover ->
PythonEnvironment::from_root`) for the higher level commands, but
couldn't spot a good place to test that.

Related discussion:
#1495 (comment) /
#1578 (comment)
zanieb added a commit that referenced this issue Sep 3, 2024
…ONMENT` (#6834)

Allows configuration of the (currently hard-coded) path to the virtual
environment in projects using the `UV_PROJECT_ENVIRONMENT` environment
variable.

If empty, we'll ignore it. If a relative path, it will be resolved
relative to the workspace root. If an absolute path, we'll use that.

This feature targets use in Docker images and CI. The variable is
intended to be set once in an isolated system and used for all uv
operations.

We do not expose a CLI option or configuration file setting — we may
pursue those later but I see them as lower priority. I think a
system-level environment variable addresses the most pressing use-cases
here.

This doesn't special-case the system environment. Which means that you
can use this to write to the system Python environment. I would
generally strongly recommend against doing so. The insightful comment
from @edmorley at
#5229 (comment)
provides some context on why. More generally, `uv sync` will remove
packages from the environment by default. This means that if the system
environment contains any packages relevant to the operation of the
system (that are not dependencies of your project), `uv sync` will break
it. I'd only use this in Docker or CI, if anywhere. Virtual environments
have lots of benefits, and it's only [one line to "activate"
them](https://docs.astral.sh/uv/guides/integration/docker/#using-the-environment).

If you are considering using this feature to use Docker bind mounts for
developing in containers, I would highly recommend reading our [Docker
container development
documentation](https://docs.astral.sh/uv/guides/integration/docker/#developing-in-a-container)
first. If the solutions there do not work for you, please open an issue
describing your use-case and why.

We do not read `VIRTUAL_ENV` and do not have plans to at this time.
Reading `VIRTUAL_ENV` is high-risk, because users can easily leave an
environment active and use the uv project interface today. Reading
`VIRTUAL_ENV` would be a breaking change. Additionally, uv is
intentionally moving away from the concept of "active environments" and
I don't think syncing to an "active" environment is the right behavior
while managing projects. I plan to add a warning if `VIRTUAL_ENV` is
set, to avoid confusion in this area (see
#6864).

This does not directly enable centrally managed virtual environments. If
you set `UV_PROJECT_ENVIRONMENT` to an absolute path and use it across
multiple projects, they will clobber each other's environments. However,
you could use this with something like `direnv` to achieve "centrally
managed" environments. I intend to build a prototype of this eventually.
See #1495 for more details on this use-case.

Lots of discussion about this feature in:

- astral-sh/rye#371
- astral-sh/rye#1222
- astral-sh/rye#1211
- #5229
- #6669
- #6612

Follow-ups:

- #6835 
- #6864
- Document this in the project concept documentation (can probably
re-use some of this post)

Closes #6669
Closes #5229
Closes #6612
@callegar
Copy link

callegar commented Sep 18, 2024

When uv does go in the higher level workflow direction, I'd advocate leaving a .venv symlink or file pointing to the central location.

and

It would be nice if "uv venv" would make the link by default.
And this would be a nice solution for managing venvs centrally outside the project-folder.

Even this makes things more complex than needed if you sync to the cloud across multiple systems, because the link might need to be to different places on different machines.

If only for compatibility and ease of migration from one tool to another, I would offer the possibility to do what pdm does. That tool offers either the option to have a local .venv or the option to have venvs stored all together in a centralized place.

Note that having the venvs all stored in a single place also makes it easier to apply deduplication tools on suitable filesystems.

@PhilipVinc
Copy link
Contributor

@zanieb is this a feature you're considering for the short term, or it's not high priority?

@Cornelius-Figgle
Copy link

Just to weight in, I sometimes edit projects (mainly for college) off a SMB share on my home server. I generally prefer having .venv in the project folder as it is neater, but the venv tools for python don't seem to play nice with SMB, so the solution is to store the venvs locally. In poetry this is rather easy as you just add a poetry.toml to the project that sets in-project to false (since my global config sets it to true for most projects). Having a way to do this in uv would be very useful, as besides this uv seems an amazing tool.

@zanieb
Copy link
Member

zanieb commented Dec 17, 2024

@Cornelius-Figgle Does a symlink work for you, e.g., .venv -> <somewhere else>?

@sitic
Copy link

sitic commented Jan 31, 2025

I'm long-time virtualenvwrapper user and miss the convenience of activating central, named, virtual environments with a quick workon command.

I wrote a small bash/zsh script called uv-virtualenvwrapper that provides core virtualenvwrapper-like functionality for uv with tab completion. mkvirtualenv <name> to create (arguments passed to uv venv), workon <name> to activate, rmvirtualenv to remove, and lsvirtualenv to list environments stored. Virtual environments are stored in ~/.virtualenvs by default and the script compatible with existing virtualenvwrapper venvs.

@Avasam
Copy link
Contributor

Avasam commented Feb 3, 2025

Just hit the need for multiple as I have a project that I test on both Windows and WSL.
I thought I could simply manually activate the .venv first to get uv to use it. But uv actively denies respecting VIRTUAL_ENV:

Image

avasam@Horus:/mnt/e/Users/Avasam/Documents/Git/AutoSplit$ uv venv .venv-linux
Using CPython 3.11.9 interpreter at: /home/avasam/.pyenv/versions/3.11.9/bin/python3
Creating virtual environment at: .venv-linux
Activate with: source .venv-linux/bin/activate
avasam@Horus:/mnt/e/Users/Avasam/Documents/Git/AutoSplit$ source .venv-linux/bin/activate
(.venv-linux) avasam@Horus:/mnt/e/Users/Avasam/Documents/Git/AutoSplit$ uv sync
warning: `VIRTUAL_ENV=.venv-linux` does not match the project environment path `.venv` and will be ignored
error: Project virtual environment directory `/mnt/e/Users/Avasam/Documents/Git/AutoSplit/.venv` cannot be used because it is not a valid Python environment (no Python executable was found)

Edit: See answer below. Not a full out-of-the-box solution. But it works as a workaround. One can use pyenv to manage different environments and manually update UV_PROJECT_ENVIRONMENT if using WSL for multiple projects.

@charliermarsh
Copy link
Member

Correct. You can use UV_PROJECT_ENVIRONMENT if you want to overwrite it, per the docs: https://docs.astral.sh/uv/concepts/projects/config/#project-environment-path.

@woutervh
Copy link

woutervh commented Feb 3, 2025

If uv would gain this functionality, I would like the option to easily create a .venv-symlink to that centralized venv-location outside the project directory.

@Nick-Hemenway
Copy link

Are there any updates on plans to support this feature? A natively supported interface similar to virtualenvwrapper would be incredible. I think it's a pretty common desire to not want a .venv folder to be included in OneDrive/iCloud file syncing. The ability to re-use environments across different folders would be a huge plus as well.

@Avasam
Copy link
Contributor

Avasam commented Feb 11, 2025

For me, uv 0.5.29's --active flag resolved my need for this feature for testing though WSL:
I can manually create an environment once. Then as long as that env is activated, uv sync --activated will work to update my dependencies when in WSL.

Image

@Nick-Hemenway
Copy link

Nick-Hemenway commented Feb 12, 2025

Using the --active flag works and allows a user to add and remove packages to a centralized virtual environment using, e.g. uv add --active numpy or uv remove --active. It would be nice if there was a way (perhaps through an environment variable?) to turn the --active flag on by default so that one doesn't have to remember to type the --active flag on every time they invoke uv to update their package/environment dependencies.

@PhilipVinc
Copy link
Contributor

I had an idea about this issue.
If I understand correctly, a counter-argument to storing virtual environments in a centralised location are other tools would not work correctly because they do not know where to look for the venv directory.
However, there are several situations where this is necessary (again, storing projects in dropbox/cloud services, or working in HPC environments where projects and virtual environments should be stored on different filesystems).

To solve the conundrum, why not have an option to store the .venv directory in a centralised location, as to address this issue, but symlink it to the current project directory, as to allow discoverability by all other tools?

Every time we run uv sync --depot XYZ or ùv run --depot XZY the symlink would be recreated if stale.

I believe this would not introduce any issue?

@PhilipVinc
Copy link
Contributor

To provide some more context to this idea:
I fully buy in uv's point of view that activating environments is an anti-pattern and that the best approach is never to have to do that.

However, right now, if we are unable to store the .venv in the current directory for whatever reason (again, HPC setups usually forbid or prevent this, or cloud setups...) the current workaround, of declaring VIRTUAL_ENV + --active or UV_PROJECT_ENVIRONMENT is equivalent to activating an environment.

What I propose above, to have an env variable UV_GLOBAL_DEPOT=XYZ which automatically stores the .venv in the global depot. The precise path could be something like $UV_GLOBAL_DEPOT/$(hash project_path)
However, by simlinking the venv to the project directory, the fact that the venv is stored elsewhere becomes unimportant for all existing tooling.
This would allow to benefit from uv's opinionated 'no activation workflow' on systems where the current behaviour cannot be used.

@john-hen
Copy link

I believe this would not introduce any issue?

I've tried this approach for a bit. I have my own little shell/batch script that manages my "central" venvs for me, on Windows, Mac, WSL, and HPC. One issue this introduces is that Git does not follow symlinks, so it won't see the .gitignore file that UV places there for our convenience. Plus, creating symlinks on Windows (usually) requires admin privileges, so UV would need to learn how to request a UAC prompt.

@PhilipVinc
Copy link
Contributor

indeed, you'd have to add a .gitignore entry to the gitignore in the root directory in that case...

@jatonline
Copy link

jatonline commented Feb 14, 2025

@PhilipVinc

I believe this would not introduce any issue?

In the discussion above, @ResRipper and @matterhorn103 talked about one motivation for having the virtual environment outisde the project directory being to stop cloud sync tools from uploading the .venv to the cloud.

If I've read the above discussion correctly, one suggestion has been to place a symlink at .venv to some central depot directory. I'm not sure how well-known this is, so thought it might be helpful to add that: on some systems (e.g. recent MacOS and Dropbox), if you put symlink inside the cloud sync folder, this gets followed by the sync client and so get uploaded to the cloud as well. (So I'm thinking this might not fix some of the originally stated issues?)

I've noticed this behaviour when using detached environments with pixi, too.

@callegar
Copy link

callegar commented Feb 14, 2025

Maybe for your case it would be nice if uv could support either .venv be the virtual environment directory or it be a file (I think that git does something similar on platforms that do not support symlinks well). In the latter case, it could contain a line such as linkto = <location>. So it could work equivalently to a symlink, with the additional advantages of: (i) not being incorrectly followed by poor synchronization tools; and (ii) being easier on platforms where symlinks are not so well supported. Furthermore, this could be enhanced by letting <location> be a string where some keyworks get expanded. For instance {hostname}, {project_name}, {path_hash} (being the hash of the project path), or {python_version}. This would let one easily avoid clashes (e.g. similarly to what pdm does for centralized venvs).

@DanCardin
Copy link
Author

However, by simlinking the venv to the project directory, the fact that the venv is stored elsewhere becomes unimportant for all existing tooling.

Most existing tooling doesn't care about the path of your venv as it is, they look for VIRTUAL_ENV. So, i think all all a symlink does is work around the lack of a uv activate command (and corresponding shell integration) which sets that env var to the central location determined by uv.

uv's preference against venv activation makes sense certainly for uv commands themselves; i think it starts to get more annoying when i'm forced to uv run pytest instead of just pytest; and i think it devolves from there with e.g. vim where it needs to be in the env for editor tooling to function (or else be able to detect the venv a-la vscode).

but i'm somewhat pessimistic on whether any of this will get dealt with in any case, due to their stance on respecting VIRTUAL_ENV in the first place...

$UV_GLOBAL_DEPOT/$(hash project_path)

per my OP, I would very much prefer a solution that allows me to have multiple named venvs per project, typically for testing multiple versions of python in quick sequence).

so while it doesnt need to be my suggestion, i think $some-root-uv-path/relative/path/to/project/{venv-name} (where venv-name defaults to .venv.{python-version}) makes a lot of sense, such that uv run -p 3.11 "just works" and selects the correct venv for the selected version of python.

@francois-rozet
Copy link

francois-rozet commented Feb 15, 2025

If you use bash, you can setup a command that looks for virtual environments at a global location (e.g. ~/.venvs) and activates them based on the env name. For example, copy-paste the following in your .bashrc and now (after restarting the shell) the aa command allows to activate a globally stored env, with auto-completion!

aa() {
    if [[ $# -eq 0 ]]
    then
        . .venv/bin/activate
    else
        . ~/.venvs/$1/bin/activate
    fi
}

export -f aa

_activate_venv_autocomplete() {
    if [[ $COMP_CWORD == 1 ]]
    then
        local word=${COMP_WORDS[COMP_CWORD]}
        local envs=$(ls -1 $HOME/.venvs)
        COMPREPLY=($(compgen -W "$envs" -- "$word"))
    else
        COMPREPLY=()
    fi
}

complete -F _activate_venv_autocomplete aa

When you create a new env with uv, remember to store it in ~/.venvs.

uv venv ~/.venvs/myenv
aa myenv

@PhilipVinc
Copy link
Contributor

My point is that uv’s mental model is to deprecate activating environments, as it can lead to problems and mismatches.

I would like to be able to use uv’s workflow in setups where I normally can’t, such as HPC clusters.
Right now it’s impossible.

@francois-rozet
Copy link

I think we are very far from being able to deprecate activating environments. Might as well make my current life easier 😉

@astrojuanlu
Copy link

Off topic

I think we are very far from being able to deprecate activating environments.

I don't think so. uv run already works and takes care of everything under the hood. It's an abstraction, so in the future it could decouple itself from in-tree venvs. Or somebody else could create a uvc run that uses centralized locations. The point is to stop doing source .venv/bin/activate or whatever OS-specific command you need (see also #1910). In Node.JS and Rust lands folks are used to doing npm run and cargo run, I don't see why we can't adopt that in Python. For that though, people need to internalise this abstraction, and documentation and instructional materials need to be updated accordingly.

@francois-rozet
Copy link

francois-rozet commented Feb 16, 2025

Off topic

For that though, people need to internalise this abstraction, and documentation and instructional materials need to be updated accordingly.

Unfortunately, technology changes faster than (most) people. I already use uv run, it is great. But I also know people that are still using Python 2.7 and 3.6. And on HPC clusters, the norm is still (and will likely remain for a while) to create an env, activate it, install your dependencies and run your code. Improving that workflow with uv venv and uv pip is already a huge step. Deprecating activating the env can come later.

@callegar
Copy link

Off topic

It is a matter of what you are used to and what you are doing. Yes, in many cases not having the shell hold the a state can be advantageous. Yet, depending on what you do typing uv run every time might end up more verbose than typing uv venv --activate once (should anything like this get supported or . .venv/bin/activate otherwise). Furthermore, there are really cases when you want to use uv, not to develop a project, but just to have a well defined virtual environment to play in (think working on notebooks that need a specific environment). In the latter case, you may really prefer having the shell holding the state to assure that you can cd to different places maintaining the state.

@mil-ad
Copy link

mil-ad commented Feb 17, 2025

I tried setting UV_PROJECT_ENVIRONMENT to something like export UV_PROJECT_ENVIRONMENT=/scratch/$USER/venvs/${PWD#$HOME}" for all users.

This mostly works but I'm one issue I'm having is that uv venv ignores UV_PROJECT_ENVIRONMENT #10807

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or improvement to existing functionality projects Related to project management capabilities virtualenv Related to virtual environments
Projects
None yet
Development

No branches or pull requests