PyEnv + Poetry: Poetry does not use correct Python versions

[S-UP]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 12.01

• Poetry version: 1.1.13

Issue

I find it extremely challenging, i.e. impossible, to get poetry to use the correct Python version. I am using PyEnv to manage Python.

which python
/Users/XXX/.pyenv/shims/python

pyenv versions
  system
* 3.8.12 (set by /Users/XXX/projectA/.python-version)
  3.9.7

python -V
Python 3.8.12

BUT within projectA:

poetry shell

python -V
Python 3.10.2

Even though:

[tool.poetry]
name = "ABC"
version = "0.1.0"
description = ""
authors = ["XXX"]

[tool.poetry.dependencies]
python = "3.8.12"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

I have zero clue what I am doing wrong. Reading a couple of online resources did not help my understanding much. However obvious this is for core users, it is extremely frustrating and counterintuitive for me.

How to set this up correctly?

See also:

• https://blog.karnwong.me/python-venv-management/
• https://lifesaver.codes/answer/poetry-using-the-wrong-python-version-not-related-to-pyenv-655

[n8sty]

Try poetry env use python3.8

[caniko]

Try poetry env use python3.8

This won't work if you use pyenv. Are you using apt?

[davehowell]

Do you have some other Python installed, e.g. from homebrew? The version you have 3.10.2 is not one of the versions that you have with pyenv so must have come from somewhere else.

Do you have PYTHONPATH set? I've had similar problem where that env var was set to a different version and the pyenv local as set by .python_version was being ignored.

[finswimmer]

Creating a venv with the current active Python on poetry install had only ever worked, when Poetry was installed via the get-poetry.py script. This was a side effect of the fact, that Poetry itself runs then with currently active Python version.

When Poetry was installed in any other way (install-poetry.py, pip, pipx) it will use the Python that was used during the installation, because Poetry now runs in its own environment and have no idea about your current active Python.

One can point Poetry to a specific Python binary that should be used to create the venv, by running poetry env use /path/to/python once per project before the venv creation. In case of pyenv this can look like this:

pyenv shell 3.8.12
poetry env use python3.8
poetry install

You can not use python as an argument for poetry env use here, because its forwarded to the underlying virtualenv which understand it as the system python and not the currently activated python.

With the next alpha release there will be an option experimental option virtualenvs.prefer-active-python (See #4852). Setting this to true, one should be able to switch to the desired Python version by pyenv before the venv creation and Poetry will than pick it up - regardless of the installation method.

fin swimmer

[jacopok]

I believe I recently encountered a very similar issue - while using tox with pyenv,
the python version was correct (python --version changed while cycling through the options)
while the virtual environment created by poetry was not (poetry run python --version was constant).

A workaround that worked for me was to deactivate the poetry virtualenv:

poetry env use system

If I understand correctly, what was happening was that poetry was using its virtualenv by default even though tox/pyenv should have told it not to; so,
the option mentioned by @finswimmer seems like it should fix this as well.

I apologize if I'm saying something obvious, if this was explained before it was not really clear to me before some debugging.

[thoughtfuldata]

@finswimmer Your solution doesnt work with a windows computer. I am using pyenv-win and even when I successfully create a shell it still installs the previous python version. I am attempting to go from python 3.9.6 to 3.9.11

These are my steps:

1. pyenv shell 3.9.11
2. poetry env use python ( i do check that python points to the 3.9.11)
3. poetry install

poetry installs 3.9.6 instead of 3.9.11

I am using poetry version 1.1.12( I just realized I have an issue with updating to 1.1.13), Windows 11, Pyenv-win 2.64.11

[finswimmer]

Hello @thoughtfuldata,

poetry env use python

unfortunately this will not work. The python is passed to virtualenv under the hood and virtualenv interpreted this as the default python. Instead run poetry env use python3.9 if you switched to a python3.9 via pyenv before.

Furthermore poetry 1.2.0b1 is out which now contains the experimental option virtualenvs.prefer-active-python as explained in #5252 (comment). You can give it a try.

[shaoye]

@finswimmer Your solution doesnt work with a windows computer. I am using pyenv-win and even when I successfully create a shell it still installs the previous python version. I am attempting to go from python 3.9.6 to 3.9.11

These are my steps:

1. pyenv shell 3.9.11
2. poetry env use python ( i do check that python points to the 3.9.11)
3. poetry install

poetry installs 3.9.6 instead of 3.9.11

I am using poetry version 1.1.12( I just realized I have an issue with updating to 1.1.13), Windows 11, Pyenv-win 2.64.11

I use

pyenv shell 3.9.11
poetry env use $(pyenv which python)
poetry install

and it worked

[claui]

I use

pyenv shell 3.9.11
poetry env use $(pyenv which python)
poetry install

and it worked

Good point @shaoye!

An alternative workaround, which is also platform-independent:

pyenv shell 3.9.11
pyenv exec pip install poetry
pyenv exec poetry install

Explanation:

• The pyenv exec pip install poetry command will install a Poetry binary to the venv that Pyenv uses internally for your target Python. Pyenv is going to reuse that internal venv for other projects that use the same Python version. So it’s not really a waste of space.
• The pyenv exec poetry install command is needed once per project. It will create the project-specific venv with the correct Python version.

[sumeeterito]

This solution does not work for me. I am on poetry 1.5.1 and when i run

• pyenv shell 3.7.16
• poetry env use $(pyenv which python)
• poetry install
• poetry add scikit-learn

I get

Error: 'numpy' must be installed before running the build.

  at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/chef.py

you can see that the poetry environment is referencing a python 3.8.

even poetry run python --version yields 3.7.16; but still it references python 3.8

[sthysel]

This usually works for me

pyenv shell 3.10.2
poetry env remove --all
poetry env use $(which python)
poetry env info 

[jmcgrath207]

Using the pyshell won't work in an automated way. I did something like this in my bash script.

poetry env use "${HOME}/.pyenv/versions/${python_version}/bin/python3"

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.