Poetry after 1.1.9 fails to recognize conda environments as valid

[thurse93]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10 20H2
• Poetry version: 1.1.9

Issue

Since poetry version 1.1.9 poetry does not recognize valid virtual environments created by conda.

We use conda (via tox-conda) to test our libraries with different versions of python (since we're in a corporate network disconnected from the internet we have to rely on repositories for the python interpreters). Minimal tox example to reproduce the problem:

tox.ini

[tox]
requires = tox-conda
envlist = debug

[testenv]

[testenv:debug]
deps =
    poetry == 1.1.8
commands =
    python -m poetry env info

Produces something like:

debug run-test: commands[0] | python -m poetry env info

Virtualenv
Python:         3.7.11
Implementation: CPython
Path:           C:\path\to\project\.tox\debug
Valid:          True

Whereas

[tox]
requires = tox-conda
envlist = debug

[testenv]

[testenv:debug]
deps =
    poetry == 1.1.9
commands =
    python -m poetry env info

Leads to a broken environment (in the eyes of poetry):

debug run-test: commands[0] | python -m poetry env info

Virtualenv
Python:         3.7.11
Implementation: CPython
Path:           C:\path\to\project\.tox\debug
Valid:          False

We think this is because conda create does not put the python executables inside the Scripts directory like virtualenv does. Instead, they are placed in the root directory of the virtualenv folder.

If we understood #4507 right, poetry 1.1.9 expects all binaries in the Scripts (or bin) Directory of the virtualenv directory.

[zEdS15B3GCwq]

I'm also having the same issue. Tested poetry 1.1.9 through 1.1.11 (latest).

My setup is the following:

• Windows 10
• conda 4.10.3 (latest)
• system Python 3.9.9
• poetry installed 1.1.9 ... 1.1.11
• conda environment created with conda create -n test python=3.9 pip=21.3

pyproject.toml:

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = ">=3.9.7,<3.10"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

First, I executed poetry show just to see the output; at first this just created a virtual env, on the second run, however, poetry reported that the conda venv was broken. Then, I tried installing pynat as a test package without additional dependencies, and poetry did as if it actually installed something, but it didn't show up in the virtual environment. Command line log (real path and username were simplified; if there's any discrepancy it doesn't mean that's the reason for the problem):

(test) C:\test>poetry --version
Poetry version 1.1.9

(test) C:\test>poetry show
Creating virtualenv test-oXodMIJs-py3.9 in C:\Users\R\AppData\Local\pypoetry\Cache\virtualenvs

(test) C:\test>poetry show
The virtual environment found in C:\Users\R\Conda\test seems to be broken.
Recreating virtualenv test-oXodMIJs-py3.9 in C:\Users\R\AppData\Local\pypoetry\Cache\virtualenvs\test-oXodMIJs-py3.9

(test) C:\test>poetry add pynat
The virtual environment found in C:\Users\R\Conda\test seems to be broken.
Recreating virtualenv test-oXodMIJs-py3.9 in C:\Users\R\AppData\Local\pypoetry\Cache\virtualenvs\test-oXodMIJs-py3.9
Using version ^0.6.2 for pynat

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing pynat (0.6.2)

(test) C:\test>conda list
# packages in environment at C:\Users\R\Conda\test:
#
# Name                    Version                   Build  Channel
ca-certificates           2021.10.26           haa95532_2
certifi                   2021.10.8        py39haa95532_0
openssl                   1.1.1l               h2bbff1b_0
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
python                    3.9.7                h6244533_1
setuptools                58.0.4           py39haa95532_0
sqlite                    3.36.0               h2bbff1b_0
tzdata                    2021e                hda174b7_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
wheel                     0.37.0             pyhd3eb1b0_1
wincertstore              0.2              py39haa95532_2

(test) C:\test>poetry remove pynat
The virtual environment found in C:\Users\R\Conda\test seems to be broken.
Recreating virtualenv test-oXodMIJs-py3.9 in C:\Users\R\AppData\Local\pypoetry\Cache\virtualenvs\test-oXodMIJs-py3.9
Updating dependencies
Resolving dependencies...

Writing lock file

No dependencies to install or update

I got the same result with poetry versions 1.1.9 through 1.1.11.

UPDATE:

OP was right about the missing python.exe in the scripts folder. With symlinks for python.exe and python39.dll in the conda environment's scripts folder, poetry works without issues:

(test) C:\test>poetry --version
Poetry version 1.1.11

(test) C:\test>poetry show

(test) C:\test>poetry add pynat
Using version ^0.6.2 for pynat

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing pynat (0.6.2)

(test) C:\test>conda list
# packages in environment at C:\Users\R\Conda\test:
#
# Name                    Version                   Build  Channel
ca-certificates           2021.10.26           haa95532_2
certifi                   2021.10.8        py39haa95532_0
openssl                   1.1.1l               h2bbff1b_0
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
pynat                     0.6.2                    pypi_0    pypi
python                    3.9.7                h6244533_1
setuptools                58.0.4           py39haa95532_0
sqlite                    3.36.0               h2bbff1b_0
tzdata                    2021e                hda174b7_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
wheel                     0.37.0             pyhd3eb1b0_1
wincertstore              0.2              py39haa95532_2

(test) C:\test>poetry remove pynat
Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing pynat (0.6.2)

(test) C:\test>conda list
# packages in environment at C:\Users\R\Conda\test:
#
# Name                    Version                   Build  Channel
ca-certificates           2021.10.26           haa95532_2
certifi                   2021.10.8        py39haa95532_0
openssl                   1.1.1l               h2bbff1b_0
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
python                    3.9.7                h6244533_1
setuptools                58.0.4           py39haa95532_0
sqlite                    3.36.0               h2bbff1b_0
tzdata                    2021e                hda174b7_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
wheel                     0.37.0             pyhd3eb1b0_1
wincertstore              0.2              py39haa95532_2

(test) C:\test>dir C:\Users\R\Conda\test\Scripts
[...]
 Directory of C:\Users\R\Conda\test\Scripts

2021. 11. 24.  16:33    <DIR>          .
2021. 11. 24.  16:33    <DIR>          ..
2021. 09. 17.  07:13                74 2to3-script.py
2021. 09. 17.  07:13            41 984 2to3.exe
2021. 09. 17.  07:13                72 idle-script.py
2021. 09. 17.  07:13            41 984 idle.exe
2021. 11. 24.  15:37               205 pip-script.py
2021. 06. 30.  14:24            74 752 pip.exe
2021. 11. 24.  15:37               205 pip3-script.py
2021. 06. 30.  14:24            74 752 pip3.exe
2021. 09. 17.  07:13                57 pydoc-script.py
2021. 09. 17.  07:13            41 984 pydoc.exe
2021. 11. 24.  16:26    <SYMLINK>      python.exe [..\python.exe]
2021. 11. 24.  16:27    <SYMLINK>      python39.dll [..\python39.dll]
2021. 11. 24.  14:05               192 wheel-script.py
2021. 06. 30.  14:24            74 752 wheel.exe

TL;DR: poetry >=1.1.9 doesn't work with conda environments, and cannot install packages into them as it expects python.exe to reside in the enviroment's scripts folder.

[troll-os]

Same thing here, tried to install with 1.1.11 and when I import my lib in a python shell I get error on imports of libs such as Pandas which obviously are in my pyproject.toml

[zljubisic]

I also experienced this problem.

I created a new conda environment, activate it, and installed poetry in it with pip poetry install.
Poetry add/remove works with no issues, pyproject.toml is correctly updated, but no packages in conda environment.

From time to time (for example with poetry show) I get:

The virtual environment found in C:\Users\...\miniconda3\envs\test_poetry seems to be broken.
Recreating virtualenv test-poetry-Um856anE-py3.9 in C:\Users\...\AppData\Local\pypoetry\Cache\virtualenvs\test-poetry-Um856anE-py3.9

For example, I installed uvicorn (poetry add uvicorn), but no uvicorn in conda environment, nor I can starti it with poetry run uvicorn.

Do we have any progress with this issue?

[zEdS15B3GCwq]

Issue persists in new release 1.1.12.

[zljubisic]

So the only workaround is: "With symlinks for python.exe and python39.dll in the conda environment's scripts folder, poetry works without issues", as it is stated above?
Does it work in windows as well?

[zEdS15B3GCwq]

So the only workaround is: "With symlinks for python.exe and python39.dll in the conda environment's scripts folder, poetry works without issues", as it is stated above? Does it work in windows as well?

Windows is the indicated OS in both the OP's and my posts. In fact, the only time I briefly tried miniconda + poetry on Ubuntu 20.04, there were no such issues. On Windows, symbolic links work for me - of course, it's not optimal that for each environment they need to be created manually.

[zljubisic]

Thanks

[finswimmer]

OMG, python is located under $CONDA_PREFIX and pip under $CONDA_PREFIX\Scripts 🤦 Let's see how to handle this ...

[finswimmer]

I've prepared more or less proof-of-concept PR's. Please be so kind to test it:

Poetry 1.1: #5008
Poetry 1.2: #5007

You can easily install it using pipx, e.g.:

pipx install --suffix=@conda11 "poetry @ git+https://github.com/finswimmer/poetry.git@ports/issue-4566-broken-conda"
pipx install --suffix=@conda12 "poetry @ git+https://github.com/finswimmer/poetry.git@issue-4566-broken-conda"

And then run by using poetry@conda11 respectively poetry@conda12

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.