Support multiple python versions for in-project virtual environment management

[abn]

The current scenario when virtualenvs.in-project = true is specified, is that we default to creating .venv as the virtual environment. This needs to be improved to support multiple versions of python.

Ideally we will end up having a structure similar to the one below.

.
└── .venv
    ├── 3.7
    ├── 3.8
    └── 3.9

Concensus Required

• Directory structure and naming.
• Should we default to this behaviour or make it opt-in only?
• If we want to default to this behaviour, should we start by opt-in first, and then opt-out?
• How should existing environments be handled? We can detect this and set project level config.

Tasks

• Drive concensus, approved by members of @python-poetry/core
• Implement new structure
• Implement test cases that validate new environment creation(s), gracefull handling of existing environments, compatibility with existing env sub-commands.

Related issues: #2908

[finswimmer]

I would like to keep the current behavior for virtualenvs.in-project = true. It is common to have the the venv in a folder called .venv. Also some IDE can then autodetect the venv.

But I like the idea to have multiple venvs in the projects folder itself. We can come already quite close to it by setting virtualenvs.path = ".virtualenvs". But this only works if you than run all your poetry commands at the same folder level as this folder. So if we could set e.g. virtualenvs.path = "{project-dir}/virtualenvs" and project-dir get's interpreted as the folder containing the pyproject.toml, we are already done.

Of course we could also hardcode the folder within the project's root where the different venv gets installed.

[artorias111]

Hi! I can take this up if no one's working on it

[finswimmer]

For sure @artorias111. We always appreciate contribution.

But we need to first come to a consensus on how it should look like, as @abn described in his post.

fin swimmer

[KGB33]

What if poetry were to use symlinks to change the python version while keeping the .venv folder?

./
├── .py_versions/
│  ├── .py3.8/
│  └── .py3.10/
├── src/
├── test/
├── .venv -> .py_versions/.py3.8
└── pyproject.toml

Where .py3.8/ and .py3.10/ are virtualenvs for their respective python versions.

One downside of using a symlink is that rm -r .venv won't remove the virtualenv.

[abn]

@finswimmer I agree. I think overloading .venv is probably not the best idea. I also agree that we should support {project-dir} or {project-root} in cofiguration templating. This might be another issue altogether.

@artorias111 feel free to pick this up. As mentioned, first task is to drive concensus and document it. Then implementation.

@KGB33 I am not sure if there already exists any standards for this. If a standard exists, we should adhere to that or see if any conventions already exists out there. I thought about the symlinks approach, but I am not too keen on it because it is not really reliable across platforms and filesystems. For now I think we should leave that part of it out of scope for this issue.

Along the lines of what @KGB33 mentioned, we might also consider a .poetry directory at the porject root. Similar to how tox environments are managed. This will also allow us to use project local env.toml files and afford us opportunities to maintain other poetry specifics in there. It would have been great if python community agreed upon something like .python so that tools coupld store cache there including virtual environemnts - if anyone wants to write a PEP :).

[ericriff]

I'm working around this limitation by setting in-project to false and the cache folder to .poetry. That way I moved the whole cache to the root of my project and I end having an arbitrary number of environments inside my project.
The drawback is that relative paths for the cache is broken for python3.7(+?)
#3049

[ericriff]

I'm working around this limitation by setting in-project to false and the cache folder to .poetry. That way I moved the whole cache to the root of my project and I end having an arbitrary number of environments inside my project.
The drawback is that relative paths for the cache is broken for python3.7(+?)
#3049

I closed the PR that had a fix for having chache folder as a relative path since it has undesired behavior. Looking forward for this changes!
We relly on dev-containers so we like to have everything in the root of our project, so we don't have to mount external paths. Setting in-project to True is a big limitation since we can only have one environment ready to go at the time.

[luismojena]

I don't know if it's on purpose, but I use this command export POETRY_VIRTUALENVS_PATH="$PWD/.venvs", and what @abn is proposing happens naturally