Default virtualenv location and pipenv shell problems

[jungornti]

Hello, i have noticed that pipenv creates virtualenv in new directory, which is highly unacceptable for our company, because of our project deployment tools are targeting .venv in project directory. How do we configure that? Also, hiding virtualenv on .local/share/virtualenv looks like bad practice:

1. you never know if virtualenv you need exists
2. system gets polluted with different virtualenvs
3. you now need to manually handle virtualenvs
4. it is harder to work with if you need to recreate virtualenv manually — this is just not obvious.

Virtualenv location: /home/asyncee/.local/share/virtualenvs/project

And secondly, i get this error after running pipenv shell:

Spawning environment shell (/bin/zsh).
ERROR: The virtualenv hasn't been activated correctly.
Either the env is corrupted (try running `pew restore env`),
Or an upgrade of your Python version broke your env,
Or check the contents of your $PATH. You might be adding new directories to it
from inside your shell's configuration file.
In this case, for further details please see: https://github.com/berdario/pew#the-environment-doesnt-seem-to-be-activated

Yesterday (before new location) everything was working fine — my python installation did not change.

[kennethreitz]

no plans for making it configurable.

it sounds like your shell is not properly configured.

[kennethreitz]

added PIPENV_VENV_IN_PROJECT mode for classic operation. Not released yet.

[kennethreitz]

this will also restore your shell functionality, although i would recommend fixing your shell configuration.

[jungornti]

Thank you very much! I figured out that in new version you did use pew to manage virtualenv and that is great, but pew is not supporting pyenv, which i use, and this is what causes the problem. I decided to refuse of pyenv in favour of pew, but this issue may target all of pyenv users.

[kennethreitz]

released!

[pauleveritt]

As a note, now that the original "put virtualenv in project dir" default behavior has been changed (sadly), the Pipenv README should be changed. It shows a virtualenv getting created in the project dir.

[nateprewitt]

Hey @pauleveritt, just to be clear, the virtualenv in project dir functionality is still available but now requires an environment variable PIPENV_VENV_IN_PROJECT to activate. Placing this in your bash profile will enable this functionality as the default.

I can only find one reference to a venv in the local directory and I don't know if I'd consider that terribly misleading. We could change the path displayed but I'm not sure if that provides any significant improvement in understanding. Each project will tell you where it's installed at initialization and is easy to find with pipenv --where.

[pauleveritt]

Yep, I gave that env var a try and it worked. However, it’s no longer the default behavior, so I thought I’d note that the home page is out of date in that regard. I agree it is a minor detail, and it’s likely only me that notices it, because I’m disappointed at the change of default. But that’s an emacs vs. vi argument that doesn’t need to be re-opened. —Paul

…

On Feb 24, 2017, at 9:42 AM, Nate Prewitt ***@***.***> wrote: Hey @pauleveritt <https://github.com/pauleveritt>, just to be clear, the virtualenv in project dir functionality is still available but now requires an environment variable PIPENV_VENV_IN_PROJECT to activate. Placing this in your bash profile will enable this functionality as the default. I can only find one reference to a venv in the local directory and I don't know if I'd consider that terribly misleading. We could change the path displayed but I'm not sure if that provides any significant improvement in understanding. Each project will tell you where it's installed at initialization and is easy to find with pipenv --where. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/kennethreitz/pipenv/issues/178#issuecomment-282307779>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbx5jdMB_3oEnMZ9Zw0iSWqeAgtD5DVks5rfuxngaJpZM4LzhjH>.

[nateprewitt]

So scratch that on pipenv --where, I guess we never updated that to include the separate venv location. I'll look into getting that included. If you feel strongly that there's a useful rewrite for the README, I'd be happy to hear suggestions or look over a PR :)

[pauleveritt]

I think at the moment I'm only proposing a change of:

$ pipenv install
Creating a virtualenv for this project...
...
No package provided, installing all dependencies.
Virtualenv location: /Users/kennethreitz/repos/kr/pip2/test/.venv

...to show the virtualenv location at /Users/kennethreitz/.local/share/virtualenvs etc.

It would have helped me if it said, instead of:

Automatically creates a virtualenv in a standard location.

...perhaps in a central location, but configurable.. But again, it's likely only me that is bugged by the change in policy. I spent quite a while looking for a discussion about the rationale, but couldn't find it. Still, it's a fast-moving project designed to have opinions, and nobody owes me squat. :)

[nateprewitt]

This was a design change in 3.3.0 when we moved to pew which uses a more centralized management scheme for virtualenv projects. It was an initiative that Kenneth kicked off, so I'm not sure if there was much (public) discussion about reasoning.

I've updated the path in the README and I'll run rewording up the flagpole to @kennethreitz, if he has opinions here.

Thanks for checking in on this @pauleveritt :)

[madhavajay]

Hrm, this doesn't work as expected:

$ pipenv --where

Pipfile found at /Users/myusername/Desktop/dev/projectname/Pipfile. Considering this to be the project home.

What I would expect is:
/Users/myusername/.local/share/virtualenvs/projectname

Then I can programmatically use pipenv --where with other shell commands to do things like manually linking system opencv.

[uranusjr]

@madhavajay Maybe you’re looking for pipenv --venv instead?

[nateprewitt]

I believe @uranusjr is correct. --where is intended for the location of the Pipfile, --venv will give you the location of the virtualenv.

[madhavajay]

Oh awesome thank you, that makes more sense. 👍

[jdorel]

From a system administration perspective, it makes more sense to default to store the venv in the project, because if we simply remove the project, there are no artifacts on the system. Also, it does not appear to be instinctive to know which venv are artifacts (to save space).

From a developer perspective, it makes more sense to default to store the venv in the project, because it means we can simply share the projects folder with someone else, no installation step required. Which was one of the purposes of the initial venv.

From https://virtualenv.pypa.io/en/latest/:

In all these cases, virtualenv can help you. It creates an environment that has its own installation directories [...]

About the softwares that need a central location for the venv (I must warn that I don't know all the reasons those softwares need a central location). From my point of view, those softwares could:

• ask the user to set the VARIABLE to use a central location themselves (in bashrc or equivalent)
• add the VARIABLE in the pipfile of projects they manage (I don't know if it is supported)
• support the fact that by default, venv are in the project

To conclude, I don't see any advantages (except for third party utilities) to have a central location for venv (please correct me on this). I understand that it would involve some work for third party utilities, but I think this is the kind of change that should be made on a major update.

[agates]

I just want to point out this makes installing packages from Pipfile.lock easier in docker/containers without shipping pipenv in the final build, allowing us to trivially copy only .venv and relevant source code to the final image without having to determine the package venv directory by hash/location etc.

[alexchandel]

Please add an option in the Pipfile to enable PIPENV_VENV_IN_PROJECT behavior. Requiring an environment variable defeats the purpose of the Pipfile/pipenv.

[Arcitec]

@alexchandel It doesn't defeat the purpose. The purpose isn't to share your .venv folder to other people (their machine architecture may not even support the installed packages, such as Windows vs Linux packages). So having this option in Pipfile would just be weird.

It also doesn't affect anything at all regarding docker. When you build your docker image, you can make a script that calls pipenv --venv to get the exact path to the venv folder, for manual inclusion in the docker image.

Pipenv's default behavior of putting the venv outside the project folder is used by most other virtualenv management tools, and it makes perfect sense because it discourages people from sharing the ".venv" folder with others (since it's huge and contains package versions that may not work on the other user's machine).

The Pipfile and Pipfile.lock already guarantee that every user will install the exact versions of each package. But that they will get the version appropriate for their own platform.

And I am not joking: This IS a problem. Try doing pipenv install numpy on Linux and look in your virtualenv's site-packages folder. You will find Linux .so files but no Windows .dll files at all.

Furthermore, the .venv folder contains a per-system .venv/bin/activate script which contains the exact, absolute path of your virtual environment, which breaks if you move the .venv folder to any other location. Likewise, the .venv/.project file contains an absolute path to the project folder itself, which cannot be moved to other systems. If you try using VSCode with the Python extension, it relies on the .project file path perfectly matching your project folder path, otherwise it won't find the project's Python interpreter.

The .venv folder should NOT be shared between different machines, for so many reasons. And such usage was never part of the original virtualenv design idea.

Literally the ONLY thing that's harmed by the "default to outside the project folder" behavior is that it leaves a lot of garbage on your system if you move/rename a project folder. Technically, someone could code a tool which scans all of your virtualenv folders (~/.local/share/virtualenvs/), looks at the .project file of each, and sees if the folder still exists, and tells you about the discovered orphans and asks if you want to remove them. That would actually be a good new feature for pipenv, something like a pipenv remove-orphans command. But it's kinda dangerous to do that automatically since some virtualenvs may be from disks that are not currently mounted. So it should definitely ask about each orphaned folder and default to [N/y] to avoid accidents... Either way, I'll make a new ticket with this idea.

[Arcitec]

New ticket: #4934