Request for documention on how to use poetry with multiple python versions

[SethMMorton]

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

In order to frame my request, let me describe how I anticipate some users come to use poetry (this at least was the case for me):

1. A developer realizes they need a more integrated dependency management solution than just setuptools, and discovers pipenv because it is the top hit on google.
2. They do a bit of reading, and decide that pipenv must be good because it is For Humans™ and it is (or at least was) The Officially Recommended Python Packaging Tool from Python.org. They try it out.
3. The developer inevitably attempts to do something outside the scope of what pipenv is trying to solve and then discover a) that pipenv is really only intended for application developers, and if one is a library developer you still need to use setup.py (making them feel like a second-class citizen), and/or b) the responses to issues filed for pipenv are usually either rude or dismissive.
4. They look for an alternative, and find poetry. There is no rudeness in the issues! The tool actually tries to help all developers! Huzzah!

For people like myself that came to poetry by way of pipenv, the approach to managing virtual environments taken by poetry tripped me up for quite some time because it differs significantly from pipenv.

• In pipenv, one specifies the python version to use for virtual environment creation, and it searches the PATH and creates it for you if a match is found.
• In poetry, it creates the virtual environment from the python against which poetry was installed, or uses a pre-made .venv.

This is a fundamental difference in philosophy between the tools which leads to a fundamental difference in usage on this specific aspect of project/dependency management. Because I had been first exposed to the pipenv way, I (wrongly) assumed that poetry also worked this way and struggled to get poetry to use a python 3.5 virtual environment because I had installed poetry against python 3.6; this was especially confusing because I had set the python version in the pyproject.toml to 3.5. I did notice the Poetry and virtualenvs section, but for someone who has not yet grokked the design philosophy of poetry there is not enough information for someone used to what pipenv does to adapt to poetry's approach.

REQUEST (finally)

I think the documentation for poetry would be improved if one (or both) of the following were added to documentation:

• A section on the correct way to use poetry with multiple python versions (along with a discussion on the philosophy behind this design decision). Perhaps formalizing the discussion that occurred in WIP : Change python version at install #82? It would be useful if it address both using pyenv and also using a pre-existing .venv. This could just mean expanding the Poetry and virtualenvs section.
• A section on how to migrate to poetry from pipenv. Here, the differences in philosophy could be specifically addressed and spelled out so that new users do not get frustrated at the first step.

It would also be helpful if this information were in the README itself, or at least linked to in the README.

[festinuz]

This issue is very relevant.

Pipenv is making a lot of fuzz at the moment, but pipenv project management is just (imho) atrocious. As a pipenv refugee myself, i found the way that poetry controls virtualenvs very frustrating and un-intuitive, and there was not a lot of info regarding it - it took me some time to stumble into poetry and virtualenvs docs page where i found an explanation.

I would really love for it to be clarified. And also honestly it would be amazing if you could set a python version of virtualenv with poetry instead of some roundabout way.

[jgirardet]

After explanation in #82 i started use pyenv but at the end i would prefere choosing python version at poetry install

[festinuz]

After spending some time reading other issues where its mentioned (#9 #35 #57 #77 #82), here's a summary: A lot of people get confused about what poetry does and what it doesnt do in regard to virtualenv.

It actually is kind of an edge case where it always tries to create virtualenv and work inside of it, but it doesnt provide any support for different virtualenvs and such.

Here are my thoughts on this. Poetry can either:

1. Extend support of virtualenvs to the point that user could use poetry as an abstraction above virtualenvs (which i would love);
2. Drop support of virtualenvs altogether in favor of user using external tools such as pew/pyenv/virtualenv to manage virtualenvs by himself;
3. Clarify documentation in regards to how poetry manages virtualenvs.

Regardles of whatever gets chosen as a solution, we should really update the documentation as it is a common point of confusion among new users (myself included).

[ciarancourtney]

Can't we just make ./venv/ to poetry what ./node_modules/ is to npm?

[UPDATE]

Just found that you can configure poetry to create virtualenv in project dir, but the path is fixed to .venv. Would be nice to be able to specify this for existing projects, would make migration seamless

See https://poetry.eustace.io/docs/configuration/#settingsvirtualenvsin-project-boolean

[SethMMorton]

I was very careful to ensure that the scope of this issue was focused on documentation only when writing it to avoid a prolonged argument on how poetry should behave. I feel as though @sdispater was pretty clear in #82 how they intended poetry to be used, so it would sadden me to see this issue become an argument on that topic rather than a constructive discussion on how to communicate the proper usage to new users.

Having said that, I am not saying that if someone has a request to change behavior they should not pursue it; rather, those requests probably belong in a different issue focused on behavior changes to poetry. If one decides to go down that route, I would suggest not only requesting that the behavior be changed, but also accompany the request with concrete examples of why that behavior would be beneficial and why the current behavior does not fit your needs.

[ciarancourtney]

@SethMMorton agreed, adhering to .venv for that use case is probably best from here on out

[sdispater]

I agree that the documentation should be clearer as to how to use Poetry to manage multiple python versions.

The idea when I started Poetry that it would work like Bundler in Ruby or NPM in JS. Basically, you would use a specific tool to manage Python versions, like pyenv, and install Poetry for each needed Python version.

Basically:

• Install pyenv
• Install a specific Python version: pyenv install 3.7.0
• Install Poetry for this Python version: pyenv global 3.7.0 && curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

This is a process similar to what exists in other languages, like those mentionned above.

The use of virtualenvs is just a side effect of the Python ecosystem but, to me, it should be completely transparent and not something the end user should have to worry about or even know is there.

It's unlikely that Poetry will integrate a Python versions management tool. This is not the purpose of a package/dependency manager and their are other tools that do it well, pyenv particularly.

That being said, the documentation should be updated with complete step by step instructions.

[festinuz]

Sorry for diverging this issue from its original direction. The above comment explains the intended workflow in a clear way. I will create a draft for documentation update in a couple of hours with a pull request unless someone wants to do it.

[SethMMorton]

@festinuz That would be great! I was planning on doing the same once I finished another project, but who knows when that will be done.

[festinuz]

I've created a #332 pull request with some groundwork for documentation regarding this topic.

But oh gee writing a normal documentation is so much harder than writing code

[vincent-ferotin]

+10 for better explanations on differences with pipenv, and recommended workflow, for newbies!

As a new pipenv refugee myself, I recently tried to use poetry for a project, installing it on Debian (stable or testing) in a new dedicated virtualenv created/managed by pew through get-poetry.py script, and haven't understood why /usr/local/virtualenvs/poetry/bin/poetry installfails with obscure message [FileNotFoundError] [Errno 2] No such file or directory: 'pip'.
Now, at least, I know from comments on #82 that I have to (probably) use pyenv (sigh! one more tool to understand and grok :-( )...

[SethMMorton]

I am cross-posting this comment from the #332 PR because it seems relevant here as well.

I like pyenv (I use it at home), but there may be an issue with recommending it as the only way to manage multiple python versions because it does not support windows (pyenv/pyenv#62) and it does not look like that will happen in the future either. Is there a cross-platform or windows-specific alternative that can be demonstrated/recommended as well?

[festinuz]

Thanks, @SethMMorton . By the way, i kinda left my PR hanging (sorry!), since the comments on said PR raised questions i cant answer for poetry - like the one you cross-posted here.

There isnt any good looking way to use poetry at the moment, to be honest. Every option feels clunky or limited in some way.

[sdispater]

@SethMMorton I agree that there is no alternative to pyenv on Windows at the moment which is an issue to have any Python version installed on a Windows system. However, this is beyond the scope of Poetry and you will need to manually install the needed Python versions.

This is a limitation of the Python ecosystem at large.

Note, however, that the new WIP installer (#378) will simplify things by requiring only one installation of Poetry. The poetry tool will then pick the currently activated Python executable to create the virtualenvs.

This will not solve the problem of the absence of a good tool on Windows to manage Python versions but it should help not having to install it multiple times.

[SethMMorton]

I just want to clarify that I did not bring up pyenv on Windows in the context of installing different Python versions - that is clearly outside of the scope of poetry.

My understanding is that pyenv is being suggested as a means to switch between already installed environments (or virtual environments), and the concern I was expressing is for that case Windows users might get the impression that poetry is not intended for them since it is suggested to use a tool that does not work on Windows.

[sdispater]

No, pyenv is only suggested for managing your main python versions. See #305 (comment) for the workflow I have in mind.

Basically, the automatic creation and detection of the virtualenvs is what poetry handles itself.

[SethMMorton]

Basically, the automatic creation and detection of the virtualenvs is what poetry handles itself.

I guess my confusion comes from the fact that poetry can only automatically create a virtualenv from the python version it was installed into (at least as I understand it). It seems that in #378 users will be encouraged to have only a single poetry installation, so within my understanding poetry would not be able to automatically create a virtualenv of any desired python environment. Is my understanding incorrect?

The heart of the reason I filed this issue was that it was unclear to handle this situation, and I was hoping to rectify that by improving the documentation. I hope you do not feel like I am badgering you with all these questions, I just want to help improve the tool!

[sirex]

I have multiple python versions already installed in the system. I was surprised, that poetry could not pick it up automatically:

> poetry install

[RuntimeError]
The current Python version (3.7.2) is not supported by the project (~3.6)
Please activate a compatible Python version.

Is there a way to specify it manually?

I have never used pyenv before and I never needed it. I just do python<version> -m venv env.

[garyo]

@sirex I think if you create a virtualenv (pythonXY -mvenv env) and then activate it (source env/bin/activate), then poetry install will work. Poetry picks up the currently active virtualenv.

[sirex]

@garyo thanks for the workaround suggestion. Still it would be nice if poetry could pick latest required version automatically without any workarounds. Of course if there is no required python version available, then it could fail.

[garyo]

I'm not sure how you'd want that to work -- should poetry install without a current virtual env install into one of the systemwide pythons? How would it find them? It could try some hack like see if there's something called "pythonXY" on $PATH, but (a) that's fragile and (b) I'm not sure poetry should ever install deps into a systemwide location, at least not without user permission.
Seems to me you should normally always use poetry install in a virtualenv, unless you're doing poetry install --no-dev for production use.
All that said, I think the new poetry env subcommand will address some of your concerns because it'll create a virtualenv for you.

[sirex]

Currently poetry install if not running under virtualenv, creates a new virtualenv automatically in ~/.cache/pypoetry/virtualenvs/<projectname>-py<vresion>.

And it creates this virtualenv using a system python. So instead of blindly taking python it could be more specific with version taking pythonX.Y according to what is specified in pyproject.toml.

[nirvana-msu]

@SethMMorton I agree that there is no alternative to pyenv on Windows at the moment which is an issue to have any Python version installed on a Windows system.

Actually, conda is (to my knowledge the only) cross-platform tool which can be used to install various python versions on Windows/Linux/macOS. It's mostly used within a scientific community so not many people are aware of it. With the silent mode installation and a bit of pain you can build seamless cross-platform provisioning and deployment. On a first glance using conda just to install Python seems like an overkill (as it's meant to manage the whole environment for you), but there's really no alternative if you need to support Windows.

[paranoidi]

In poetry, it creates the virtual environment from the python against which poetry was installed, or uses a pre-made .venv.

This EXACTLY is what is ruining my poetry adoptation at the moment. I have multiple python projects with different python versions.

I have currently in the pyproject.toml

[tool.poetry.dependencies]
python = "^3.6"

But the poetry install command will create virtualenv using Python 3.7. In what sane world is it okay to make virtualenv using completely different python version than what project declares as being dependant of. The project in question does NOT work with Python 3.7.

This is not merely a documentation issue, this is a clear violation of POLA. Either create virtualenv using correct version of python or abort (fail fast, fail hard).

Now, does multiple poetry installations for each python version sound good to anyone? Not at least to me .. this needs to be handled better.

[KyleKing]

@paranoidi with python = "^3.6" Poetry thinks that it can use any version of Python 3.6-4.0. You want to use a tilde requirementpython = "~3.6". See the README for caret / tilde

(py372) C:\Users\king.kyle\Developer\Packages\common_prStub>poetry shell

[RuntimeError]
The current Python version (3.7.2) is not supported by the project (~3.6)
Please activate a compatible Python version.

(py372) C:\Users\king.kyle\Developer\Packages\common_prStub>

Also, Poetry uses whichever Python version is currently active if you've installed poetry globally using the get-poetry script. Typically the workflow is to create a virtual environment with the desired version, then run poetry install. So you could: conda create -n py368 python=3.6.8 conda activate py368 then poetry install would create a Python 3.6.8 environment for your project

[laike9m]

I think Poetry should just use whatever environment it's in, and not creating a virtualenv.

[Make42]

See the README for caret / tilde

Not sure if it is just me, but the links don't seem to work.

[SethMMorton]

Given that #731 is implemented, merged, and documented, this solves the spirit of this issue. I will close the issue now.

[Mithrandir2k18]

In poetry, it creates the virtual environment from the python against which poetry was installed, or uses a pre-made .venv.

This EXACTLY is what is ruining my poetry adoptation at the moment. I have multiple python projects with different python versions.

I have currently in the pyproject.toml

[tool.poetry.dependencies]
python = "^3.6"

But the poetry install command will create virtualenv using Python 3.7. In what sane world is it okay to make virtualenv using completely different python version than what project declares as being dependant of. The project in question does NOT work with Python 3.7.

This is not merely a documentation issue, this is a clear violation of POLA. Either create virtualenv using correct version of python or abort (fail fast, fail hard).

Now, does multiple poetry installations for each python version sound good to anyone? Not at least to me .. this needs to be handled better.

For me this is even worse, my system python is on 3.10 but in a current project I need to use tensorflow which only supports up to 3.9. So I have

python = ">=3.8,<3.10"

Which causes all sorts of problems. My only way of fixing it seems to be to install pyenv and use that as poetry cannot downgrade automatically it seems, at least poetry install fails.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.