Poetry does not use active pyenv when creating virtual environment

[bjoernpollex]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: MacOS 10.14 Mojave

• Poetry version: 0.12.8

Issue

I have a project directory with a .python-version file like this:

3.7.1

Poetry has been installed into Python 3.6.5. According to recent comments, poetry is supposed to detect the active Python version when creating a new virtual environment, but it seems to stick with 3.6.5 all the time. To illustrate:

$ python --version
Python 3.7.1
$ poetry run python --version
Python 3.6.5

When I specify python = "^3.7" in `pyproject.toml I get an error:

$ poetry shell

[RuntimeError]
The current Python version (3.6.5) is not supported by the project (^3.7)
Please activate a compatible Python version.

shell

[bjoernpollex]

Turns out that explicitly activating the correct Python version (e.g. by running pyenv shell 3.7.1) fixes this, but I was expecting this to automatically pick up what I specify in .python-version.

[sdispater]

pyenv should pick up the .python-version. If it does not you have an issue with your configuration but this has nothing to do with Poetry.

[bjoernpollex]

I'm not sure where. When I run pyenv version it gives this output:

3.7.1 (set by ./.python-version)

I'll do some debugging - I'm having trouble locating the code where poetry detects the Python version, once I find where that is I'll know more.

[sdispater]

How did you install Poetry?

[bjoernpollex]

I've used the installer from the website - I've recently updated from 0.11.5 to 0.12.8, maybe some leftovers from the old installation? Poetry is installed into another Python version managed by pyenv (that's the 3.6.5 you see above).

[sdispater]

Possibly. Try pip uninstall poetry to see if the old Poetry installation is still there.

[bjoernpollex]

Hm, though I had done that, but it was still there. However, removing it didn't change anything (and poetry debug:info was reporting the correct version before).

[sdispater]

What is the output of poetry debug:info?

[bjoernpollex]

$ poetry debug:info

Poetry
======

 * Version: 0.12.8
 * Python:  3.6.5


Virtualenv
==========

 * Python:         3.6.5
 * Implementation: CPython
 * Path:           NA


System
======

 * Platform: darwin
 * OS:       posix
 * Python:   /Users/bjoernpollex/.pyenv/versions/3.6.5

[sdispater]

I don't know what to say. Poetry sees Python 3.6.5 from pyenv not 3.7.1. What is the output of pyenv which python?

[bjoernpollex]

$ pyenv which python
/Users/bjoernpollex/.pyenv/versions/3.7.1/bin/python

[sdispater]

That's so weird. And just to be sure: can you execute which poetry?

[bjoernpollex]

$ which poetry
/Users/bjoernpollex/.poetry/bin/poetry

[bjoernpollex]

Ok, I found out that if I remove the .python-version file in my user directory (I was using this to set the default Python version), poetry is run correctly using Python 3.7.1. I'm not sure what to make of this.

[buriy]

Guys, also see #655 -- the current behavious is a regression, version 0.12.4 worked correctly for me.

[bersace]

I confirm this bug with poetry 0.12.10:

$ poetry debug:info

Poetry
======

 * Version: 0.12.10
 * Python:  3.6.7


Virtualenv
==========

 * Python:         3.6.7
 * Implementation: CPython
 * Path:           NA


System
======

 * Platform: linux
 * OS:       posix
 * Python:   /home/bersace/.local/venvs/poetry


$ pyenv which python
/home/bersace/.cache/pyenv/versions/3.7.1/bin/python
$ python --version
Python 3.7.1
$ poetry install
                                                                             
[RuntimeError]                                                
The current Python version (3.6.7) is not supported by the project (^3.7)  
Please activate a compatible Python version.                               
                                                                             
install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

[sdispater]

@bersace As you can see the Python used is /home/bersace/.local/venvs/poetry/bin/python.

If you do not use the recommended installer Poetry is linked to the Python version it has been installed for. In this case I suspect you used pipx. That's one of the shortcomings of using any pip install derived methods.

So, this is not a bug in Poetry but a limitation of the entrypoints functionality of the Python ecosystem. I'd suggest reinstalling Poetry with the recommended installer.

[bersace]

@sdispater Yes, i installed poetry using pipsi as documented here : https://github.com/sdispater/poetry/blob/master/docs/docs/index.md#installing-with-pipsi

[bersace]

@sdispater using get-poetry.py works fine ! Thank you !

[pawamoy]

I would love to see poetry work when installed with pipsi. I'm not particularly fond of another directory in my home, and another path inserted in front of PATH.

Also, I juste tried the recommended installation method (curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python) but it does not work.

$ poetry install
                                                                              
[RuntimeError]                                                 
The current Python version (2.7.13) is not supported by the project (~3.6)  
Please activate a compatible Python version.                                
                                                                              
install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

I uninstalled poetry, re-installed it using python3.6, and it still did not work (same message).

I was able to fix this by first running pyenv shell 3.6.7, then running poetry install.

Can't poetry find my Python 3.6 executable installed with pyenv? Just like it would be found if I ran virtualenv --python python3.6? There is the python3.6 shim in my PATH so I expected it to work 😕

[bjoernpollex]

@pawamoy Poetry is intentionally not managing python versions for you, as that would unnecessarily bloat the scope of the tool. Using pyenv in the way you've described above is the intended way to set things up. You can add a .python-version-file to your project, and then pyenv will automatically select the right version for you.

[bjoernpollex]

To circle back to the original issue, I've found a fix (as described above), but had no luck in figuring out what the root cause was. I do have the feeling that it's a pyenv issue, rather than a poetry issue though. I'm going to close this for now, can always reopen if it comes up again.

[bersace]

I would love to see poetry work when installed with pipsi. I'm not particularly fond of another directory in my home, and another path inserted in front of PATH.

I second this. That would be nice if get-poetry installs poetry in e.g. ~/.local/lib/pypoetry and added a link in ~/.local/bin. That's more a matter of taste.

[garyo]

The systemd file hierarchy spec (https://www.freedesktop.org/software/systemd/man/file-hierarchy.html) recommends ~/.local/bin and ~/.local/lib for this use, so I'll second that.

[jeannei]

I was having similar issues turns out it was an issue with pyenv. I followed this to resolve the steps. Specifically I had to add the following to my .bash_profile

export PATH="/Users/username/.pyenv:$PATH"
eval "$(pyenv init -)"

[rjstretch]

@jeannei That solution does seem to work. Thanks!

[tatianafrank]

I was having the same issue. Im using pyenv to manage python versions. I was changing my python version using pyenv local <version> and recreating my virtualenv in my IDE then running poetry install. But poetry debug:info was showing the old version. Then I deleted the .python-version file as noted above and ran pyenv shell with the new version and then poetry install and it did the trick.

[ben-dyer]

Had the same issue; removing .python-version from my home directory solved it

[pawamoy]

Maybe worth to add it here: sometimes my system Python is upgraded through system-updates, and it usually breaks my Python setup. When Poetry can't find the right Python versions anymore, and everything else failed what I do is:

• remove all poetry virtualenvs (rm -rf ~/.cache/pypoetry/virtualenvs/*)
• save the list of pipx tools installed, uninstall them (including poetry, yes I install it through pipx)
• uninstall pipx
• uninstall pyenv (and the installed versions)

Then I can reinstall everything:

• install pyenv
• install Python versions through pyenv
• make them available with pyenv global
• install pipx with one of the pyenv versions
• reinstall all pipx tools (including poetry)
• recreate poetry's virtualenvs

A bit brutal, but it also allows to clean up a bit 😄

[iuriguilherme]

user@localhost:~/project$ grep -e 'poetry.dependencies' -A 1 pyproject.toml
[tool.poetry.dependencies]
python = "^3.7"

user@localhost:~/project$ whereis poetry
poetry: /home/user/.local/bin/poetry

user@localhost:~/project$ which poetry
/home/user/.local/bin/poetry

user@localhost:~/project$ head -n1 /home/user/.local/bin/poetry
#!/home/user/.pyenv/versions/3.9.2/bin/python3.9

user@localhost:~/project$ poetry run python --version
Python 3.5.3

user@localhost:~/project$ pyenv exec poetry run python --version
Python 3.5.3

user@localhost:~/project$ python3.9 -m poetry run python --version
Python 3.5.3

user@localhost:~/project$ which python
/home/user/.pyenv/shims/python

user@localhost:~/project$ python --version
Python 3.9.2

user@localhost:~/project$ pyenv global
3.9.2

user@localhost:~/project$ pyenv local
3.9.2

user@localhost:~/project$ pyenv shell
3.9.2

Added info: pipenv using a ported Pipfile from the pyproject.toml did used python3.9.2

[iuriguilherme]

Found the problem, the virtualenv had been created originally with the system python:

user@localhost:~/project$ poetry debug

Poetry
Version: 1.1.4
Python:  3.9.2

Virtualenv
Python:         3.5.3
Implementation: CPython
Path:           /home/user/.cache/pypoetry/virtualenvs/project-FrpO_8-J-py3.9
Valid:          True

System
Platform: linux
OS:       posix
Python:   /usr

user@localhost:~/project$ rm -rdv ~/.cache/pypoetry/virtualenvs/project-FrpO_8-J-py3.9/
OK

user@localhost:~/project$ poetry debug                                                                                                                                                                                                                                                                Poetry                                                                                                                                              Version: 1.1.4                                                                                                                                      Python:  3.9.2                                                                                                                                                                                                                                                                                          Virtualenv                                                                                                                                          Python:         3.9.2                                                                                                                               Implementation: CPython
Path:           NA

System
Platform: linux
OS:       posix
Python:   /home/user/.pyenv/versions/3.9.2

user@localhost:~/project$ poetry install
OK

user@localhost:~/project$ poetry debug

Poetry
Version: 1.1.4
Python:  3.9.2

Virtualenv
Python:         3.9.2
Implementation: CPython
Path:           /home/user/.cache/pypoetry/virtualenvs/project-FrpO_8-J-py3.9
Valid:          True

System
Platform: linux
OS:       posix
Python:   /home/user/.pyenv/versions/3.9.2

Some hints on #926 are misleading but I managed to fix this one.

[will-ockmore]

@bjoernpollex I'm still running into this issue with a .python-version in my user directory.

$ poetry --version
Poetry version 1.1.5

$ python --version
Python 3.9.1

$ poetry env info

Virtualenv
Python:         3.8.6
Implementation: CPython
Path:           NA

System
Platform: linux
OS:       posix
Python:   /home/will/.pyenv/versions/3.8.6

$ cat ~/.python-version
3.8.6

$ cat ./.python-version
3.9.1

pyenv shell 3.9.1 fixes it; however this should not be necessary. Ensuring pyenv's PATH setup happens after or before poetry's made no difference.

[yssource]

the same issue here.
.python-version makes it always recreating .venv under project.

[nzhwtq]

At the time I am writing this (2021.06.14), I would not advise you installing poetry using install-poetry.py. Instead, stick with the old get-poetry.py. The new one doesn't work with pyenv. It will always create the system version instead of the version pyenv specified. The old one does not have this problem.

[hute37]

poetry installed with install-poetry.py to enable poetry self update.
Same problem with pyenv python version.

It seems to work with explicit virtual env creation with:

poetry env use -- $(which python)

# Creating virtualenv ...-VfhLuGbT-py3.7 in .../.cache/pypoetry/virtualenvs
# Using virtualenv: .../.cache/pypoetry/virtualenvs/...-VfhLuGbT-py3.7

Versions:

poetry --version
# Poetry (version 1.2.0a1)

pyenv --version
#  pyenv 2.0.1-3-g1706436f

pyenv  versions  
#  system
# * 3.7.6 (set by PYENV_VERSION environment variable)
#   3.9.5

[paulmelnikow]

For the benefit of others who have landed on this thread looking for help with Pyenv-installed Python, there is an open pull request #4164 which seems to provide support for it.

[sdispater]

The new install-poetry.py script does not work the same as the old one (get-poetry.py) did. The new one will install Poetry for the current Python version only.

If you need to switch between environments, we recommend using the env use command for now or, alternatively, install Poetry in each pyenv environment.

We are currently discussing ways to improve the situation so we'll keep you posted if we ever find a better solution.

[nzhwtq]

@sdispater What is the motivation behind this switch?

[albireox]

@sdispater Is pip installing poetry for development still not recommended? I usually create a pyenv virtualenv for each project to using the script on each one of them is going to be a real annoyance, but I could see myself just pip installing poetry on each venv.

[sdispater]

@nzhwtq What motivated us to make the switch was to alleviate maintenance burden and also for security (the vendored dependencies were frozen so fixing dependencies required us to make a new release which was less than ideal).

While I understand that the convenience of having one installation of Poetry for any pyenv environment will be missed I think this is for the best.

@albireox I wouldn't say that installing via pip is recommended even though it's listed in the documentation (https://python-poetry.org/docs/master/#installing-with-pip) due to the fact that it can cause issues (see the warning in the documentation). However, your use case is already supported (and is the one I use myself). If you use pyenv-virtualenv and you create on virtual environment by project and the virtual environment is activated then Poetry will detect it automatically and use it. So for this use case nothing has changed.

[nzhwtq]

@sdispater Thank you for the explanation. For now with the new installer, if I installed it with the system python, when the system python upgraded, will poetry change its default virtual environment python version for the new project? If so, I assume the best practice has become using pyenv to change to the desired python version and install poetry rather than install it beforhand using the system python?

[bo1929]

Installing poetry with pipx was the cause of the problem for me.

[princelySid]

Just in case someone end up here after hours debugging do the usual flow then poetry env use <python version>. Should work after that. The documented flow with pyenv doesn't work any more #4317

[PiQuer]

Thanks @princelySid. That would be poetry env use $(pyenv which python) for pyenv users with an active pyenv interpreter.

[DCastile]

At the time I am writing this (2021.06.14), I would not advise you installing poetry using install-poetry.py. Instead, stick with the old get-poetry.py. The new one doesn't work with pyenv. It will always create the system version instead of the version pyenv specified. The old one does not have this problem.

Seconding this as of the time of this writing.

[garyo]

This is all getting pretty confusing. Perhaps the doc should spell out how to install & use poetry in the common python cases: (a1) regular python installer, (a2) regular installer + virtualenv, (b1) pyenv, and (b2) pyenv + virtualenv.

[crono782]

At the time I am writing this (2021.06.14), I would not advise you installing poetry using install-poetry.py. Instead, stick with the old get-poetry.py. The new one doesn't work with pyenv. It will always create the system version instead of the version pyenv specified. The old one does not have this problem.

Still same issue as of this comment.

[matbun]

At the time I am writing this (2021.06.14), I would not advise you installing poetry using install-poetry.py. Instead, stick with the old get-poetry.py. The new one doesn't work with pyenv. It will always create the system version instead of the version pyenv specified. The old one does not have this problem.

At the time of writing (2022.07.26), I can still replicate the same behaviour. Please fix it.

[darosio]

poetry == 1.1.14

1. pacman -S python-poetry does not respect pyenv (local) versions
2. pipx install poetry does not respect pyenv (local) versions
3. only the deprecated get-poetry.py does respect pyenv (local) versions

[martimors]

After many hours confusion I realize I am facing this problem :(

[neersighted]

As of 1.2, poetry config virtualenvs.prefer-active-python true exists experimentally to use a new method to detect the active Python. Please try out this setting, and start a Discussion or join Discord if you have questions about using it. If you run into issues using it (and can reproduce them in a container/clean environment/are sure you have set up pyenv correctly), please open a bug report.

I am going to lock this issue for now as it is solved, and people attempting to get support in it will be sending a lot of unnecessary notifications.