Conda compatibility

[extraymond]

Hi! I tried installing poetry alongside my miniconda env.
It's installed via the curl installer.

At first, it seems that package resolving and installing works.
When I try to run the script using:

poetry run python project/script.py

It emit the error saying that package is not found, which can be found in poetry show.

I'm not sure if this is actually conda related issue or just misuse of the CLI. Would be nice to know how I can get it working with poetry. Seems like a really nice project. Thx

[sdispater]

I don't use miniconda so I don't know if I can be of much help here.

However, you are doing it right so thre must be a bug somewhere.

Which package does it say can't be found?

And poetry show only lists what is in your lock file but does not refeclt what is installed. That's something I should change since it can be confusing.

Can you do poetry run pip freeze and tell me if your package shows up there?

[extraymond]

I suspect that has something to do with how poetry not perceiving conda as a virtualenv.

The signs:

1. When running poetry add package, it ends up in the main conda env lib path.
2. When adding it in a isolated conda env after source activate venv, it still installed packages to the main conda env lib path.
3. Non of the above action alter contents in .cache/pypoetry/virtualenvs/, if running poetry new, a new folder will popup, but the content in the subfolder of lib/python3.X remains empty.
4. When running poetry run python -V, it shows a different python than the conda one, which means its in a somewhat virtualenv or sort.

This is different compared to what poetry is doing when detecting venv with pipenv or virtualen. I think the culprit is that it's not configured or half configured to perceive conda env.

Additional info:

poetry show returns correct lists of packages, just that they are not installed in the desired direction.

Edited for additional info and 5.

[sdispater]

@extraymond I will take a look at this and get back to you.

[extraymond]

@sdispater Thx! Looking forward to using poetry.

[tngTUDOR]

Hi,
I already 👍 the issue, because I'm definitely interested in mixing well conda & poetry.
Here is what I think could help to reproduce the issue (using conda):

conda create -n testpoesia python=3.7
conda activate testpoesia
pip install poetry

poetry new rubendario

Installing a dep with conda will work fine (from condaforge):

conda install tqdm

All that works fine. Now, trying to add that package (after installing it with conda) yields:

tomas@localhost ~/rubendario                                                                                                                                                                                                                       [9:07:04] 
(testpoesia) > $ poetry add tqdm                                                                                                                                                                                                                            
Creating virtualenv rubendario-py3.7 in /home/tomas/.cache/pypoetry/virtualenvs
Using version ^4.23 for tqdm

Updating dependencies
Resolving dependencies... (2.5s)


Package operations: 8 installs, 0 updates, 0 removals

Writing lock file

  - Installing six (1.11.0)
  - Installing atomicwrites (1.1.5)
  - Installing attrs (18.1.0)
  - Installing more-itertools (4.2.0)
  - Installing pluggy (0.6.0)
  - Installing py (1.5.4)
  - Installing pytest (3.6.3)
  - Installing tqdm (4.23.4)

The intersting part (I guess) is the message from poetry that says:

Creating virtualenv rubendario-py3.7 in /home/tomas/.cache/pypoetry/virtualenvs

Meaning, that it did not identify the conda virtualenv.

@sdispater could you maybe point us to the mechanisms of poetry to detect the virtualenvs ❓ so that we can maybe try out helping

[Gwill]

I found the reason.

Because It modified the PATH in .bash_profile (or .zshrc your shell rc file) when installing Anaconda (or miniconda), something like this:

# in my .zshrc
export PATH=/Users/Gwill/anaconda/bin:$PATH

And when we run poetry shell at

https://github.com/sdispater/poetry/blob/master/poetry/utils/env.py#L295

subprocess.call the shell and shell loads rc file, the anaconda/bin path alwasys at front of venv/bin path. so the python interpreter path is wrong.

To fix this we cloud use https://github.com/pexpect/pexpect to spawn new shell and in new shell beginning to activate the venv, like pipenv do:

https://github.com/pypa/pipenv/blob/master/pipenv/shells.py#L106-L107

This can make the venv bin path at the front of PATH 😂

@sdispater

[goldslope]

I have made a tool on behalf of Eigen Technologies that integrates poetry into conda environments.

It sidesteps the virtualenv and installs the poetry packages directly into the conda environment.

We are using it to create, package, and deploy environments to clients. Any non-binary pip packages are compiled with conda's cross-compilers for maximum compatibility across OS versions/distros.

You don't even need a conda environment file, its optional; using this tool you can zip up Poetry environments into a portable cross-distro/cross-version tarball

It works for Linux and macOS only.

https://github.com/eigentechnologies/cpip

Check it out! Let me know if you have any suggestions.

[dazza-codes]

The latest releases of miniconda no longer modify the ~/.bashrc to add conda to the PATH. The latest releases use something like an /etc/profile.d/conda.sh to load bash functions (check it with type conda for example).

There is a bit of a gulf between conda packaging and pure-pip packaging, including the package repositories that have inconsistent package names and versions. Attempting to get poetry to work with conda might be mission impossible and potentially misguided. AFAICT, the aim of poetry is compatible with pure-pip packaging, but it does not aim to replace the conda package specs in environment.yml for example. conda has a larger agenda to encompass or wrap many packaging systems (apt, yum, pip) so it can manage binary shared libs (mostly c/c++ libs) that are often wrapped by python libs (e.g. a large % of the pydata stack is like this).

For my 2c, poetry is pure-pip and not conda compatible. If it tries to move into the conda domain, it could be a world of pain. If poetry detects a conda-env, it could run away and complain loudly about the misguided attempt to combine these two worlds of packaging systems. The responsibility to work with poetry might lie with conda and perhaps could include a clause in the environment.yml file to allow use of poetry as it does allow a pip clause now already. The bigger question is what to do about the conda channels vs. pypi repositories and the underlying dependency resolution engines and that might fall into the too-hard basket for a while.

[dazza-codes]

Heads up, stay away from PATH and the pyenv model until some shim problems are resolved, see pyenv/pyenv#1112

[rmorshea]

I followed these steps to install Conda and Poetry on Linux. These same steps should work for Mac as well (not sure about Windows):

1. Download Miniconda:

https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

2. Run the installer and choose where to save it. Setting up .bashrc or .bash_profile is fine.

bash Miniconda3-latest-Linux-x86_64.sh -p <CONDA-INSTALL-LOCATION>

3. Once installed open a new terminal. You should be able to run conda and get help output.

4. Install Poetry

curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

5. Open a new terminal. You should be able to run poetry and get help output.

6. Configure the Poetry's Virtualenv location:

poetry config settings.virtualenvs.path <CONDA-INSTALL-LOCATION>

7. Create and activate virtual env with conda:

conda create -n <MY-ENV-NAME> python=<PYTHON-VERSION>
conda activate <MY-ENV-NAME>

8. Go to your project directory and install from your pyproject.toml

poetry install

[dazza-codes]

Programmatic attempt at ^^ @rmorshea (good tip)

# Configure the Poetry's Virtualenv location.
# Use `conda info` and https://stedolan.github.io/jq/download/
CONDA_ENV_PATH=$(conda info --json | jq '.envs_dirs[0]')
poetry config settings.virtualenvs.path $CONDA_ENV_PATH

Still, no idea whether conda can migrate to a pyproject.toml vs. the environment.yml and how they are supposed to fully cooperate when they draw from different lib-repositories (pypi vs. conda channels). Or, vice versa, whether poetry can interpret and use environment.yml and related conda channels as a prelude to then using pyproject.toml. AFAICT, the best use of poetry with conda is to create a stand-alone poetry app (poetry-only, pip-only) that happens to use a python installation from conda (ie. use conda to manage python versions) and it does not draw from conda channels. The tempting use of both conda, with poetry as a supplement to an existing conda-env installation, is more problematic (who knows what happens with conda env update or poetry update).

[ryanfeather]

In addition to the above, I had to configure poetry to not create a new virtualenv to use the conda env::

poetry config settings.virtualenvs.path $CONDA_ENV_PATH
poetry config settings.virtualenvs.create 0

It seems like an inevitability that you are going use pip for something when using conda, so at the very least having the poetry wrapper over top is a bonus, as long as you're basically using conda as just a python vendor as @darrenleeweber describes. It seems like the "right" way to mix the two would be to create a CondaEnv and CondaInstaller that are somehow configurable within. I'm definitely not enough of a conda internals expert to attempt this though.

Given the necessity of managing scientific and numerical dependencies for python projects it sure would be nice if there really was "one-- and preferably only one --obvious way to do it."

[guidoavvisati]

On windows, I am using conda exclusively to manage different python versions and create virtual envs. Inside the virtual env I install everything via pip (akin to @darrenleeweber @ryanfeather). I can confirm that configuring poetry like @ryanfeather also works on windows. The only catch is to provide a path enclosed in quotes

poetry config settings.virtualenvs.path "C:\<path_to_anaconda3_folder>\envs"
poetry config settings.virtualenvs.create 0

[mdavis199]

None of the suggestions above (neither rmorshea nor ryanfeather) worked for me as expected.

First, on Windows 10 Enterprise, I started with a fresh copy of the latest version of MiniConda and ran conda list to get a list of the base packages:

Default Conda Package List

asn1crypto                0.24.0                   py37_0
bzip2                     1.0.6                hfa6e2cd_5
ca-certificates           2019.5.15                     0
certifi                   2019.6.16                py37_0
cffi                      1.12.2           py37h7a1dbc1_1
chardet                   3.0.4                    py37_1
conda                     4.7.5                    py37_0
conda-package-handling    1.3.10                   py37_0
console_shortcut          0.1.1                         3
cryptography              2.6.1            py37h7a1dbc1_0
idna                      2.8                      py37_0
libarchive                3.3.3                h0643e63_5
libiconv                  1.15                 h1df5818_7
libxml2                   2.9.9                h464c3ec_0
lz4-c                     1.8.1.2              h2fa13f4_0
lzo                       2.10                 h6df0209_2
menuinst                  1.4.16           py37he774522_0
openssl                   1.1.1c               he774522_1
pip                       19.0.3                   py37_0
powershell_shortcut       0.0.1                         2
pycosat                   0.6.3            py37hfa6e2cd_0
pycparser                 2.19                     py37_0
pyopenssl                 19.0.0                   py37_0
pysocks                   1.6.8                    py37_0
python                    3.7.3                h8c8aaf0_0
python-libarchive-c       2.8                      py37_6
pywin32                   223              py37hfa6e2cd_1
requests                  2.21.0                   py37_0
ruamel_yaml               0.15.46          py37hfa6e2cd_0
setuptools                41.0.0                   py37_0
six                       1.12.0                   py37_0
sqlite                    3.27.2               he774522_0
tqdm                      4.32.1                     py_0
urllib3                   1.24.1                   py37_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_0
wheel                     0.33.1                   py37_0
win_inet_pton             1.1.0                    py37_0
wincertstore              0.2                      py37_0
xz                        5.2.4                h2fa13f4_4
yaml                      0.1.7                hc54c509_2
zlib                      1.2.11               h62dcd97_3
zstd                      1.3.7                h508b16e_0

Then, I installed poetry using the "curl and pipe to python" method and verified that the 'base' environment did not change.

Then, I tried several different ways to install the basic "new" project.

A. First try:

poetry new simple
poetry install 

B. Second try (after reinstalling a fresh conda):

conda create -n simple
activate simple 
poetry new simple
poetry install

C. Third try (after reinstalling a fresh conda):

conda create -n simple
activate simple 
poetry config settings.virtualenvs.path "C:\<path_to_anaconda3_folder>\envs"
poetry config settings.virtualenvs.create 0
poetry new simple
poetry install 

Every single attempt installed the packages into the base environment.
Packages in virtual environment: conda list -n simple yielded an empty list:

Packages in 'simple' virtual environment:

# packages in environment at E:\Apps\Tools\python\conda3\envs\simple:
#
# Name                    Version                   Build  Channel

Packages in base environment: conda list -n base:

Packages installed into "base" environment:

# packages in environment at E:\Apps\Tools\python\conda3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0
atomicwrites              1.3.0                    pypi_0    pypi
attrs                     19.1.0                   pypi_0    pypi
bzip2                     1.0.6                hfa6e2cd_5
ca-certificates           2019.5.15                     0
certifi                   2019.6.16                py37_0
cffi                      1.12.2           py37h7a1dbc1_1
chardet                   3.0.4                    py37_1
colorama                  0.4.1                    pypi_0    pypi
conda                     4.7.5                    py37_0
conda-package-handling    1.3.10                   py37_0
console_shortcut          0.1.1                         3
cryptography              2.6.1            py37h7a1dbc1_0
idna                      2.8                      py37_0
importlib-metadata        0.18                     pypi_0    pypi
lemur                     0.1.0                     dev_0    <develop>
libarchive                3.3.3                h0643e63_5
libiconv                  1.15                 h1df5818_7
libxml2                   2.9.9                h464c3ec_0
lz4-c                     1.8.1.2              h2fa13f4_0
lzo                       2.10                 h6df0209_2
menuinst                  1.4.16           py37he774522_0
more-itertools            7.1.0                    pypi_0    pypi
openssl                   1.1.1c               he774522_1
pip                       19.0.3                   py37_0
pluggy                    0.12.0                   pypi_0    pypi
powershell_shortcut       0.0.1                         2
py                        1.8.0                    pypi_0    pypi
pycosat                   0.6.3            py37hfa6e2cd_0
pycparser                 2.19                     py37_0
pyopenssl                 19.0.0                   py37_0
pysocks                   1.6.8                    py37_0
pytest                    3.10.1                   pypi_0    pypi
python                    3.7.3                h8c8aaf0_0
python-libarchive-c       2.8                      py37_6
pywin32                   223              py37hfa6e2cd_1
requests                  2.21.0                   py37_0
ruamel_yaml               0.15.46          py37hfa6e2cd_0
setuptools                41.0.0                   py37_0
six                       1.12.0                   py37_0
sqlite                    3.27.2               he774522_0
tqdm                      4.32.1                     py_0
urllib3                   1.24.1                   py37_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_0
wheel                     0.33.1                   py37_0
win_inet_pton             1.1.0                    py37_0
wincertstore              0.2                      py37_0
xz                        5.2.4                h2fa13f4_4
yaml                      0.1.7                hc54c509_2
zipp                      0.5.1                    pypi_0    pypi
zlib                      1.2.11               h62dcd97_3
zstd                      1.3.7                h508b16e_0

In the third attempt, even though the commands were executed in the 'simple' virtual environment, poetry installed all packages into the 'base' environment.

[mdavis199]

Maybe poetry is assuming that the python installation has the 'virtualenvironment' package preinstalled?

I would have thought that it's installed as part of the poetry setup or initialization. It seems to work if I install the module before trying to run poetry.

[brycedrennan]

It would be nice if poetry worked inside a conda environment as long you stayed pip-only. If we can make small tweaks that make the several-step manual process documented above easier, I'm all for that.

Beyond that though, I tend to agree with @darrenleeweber that attempting compatibility between poetry and conda would be a world of pain and is perhaps not even possible.

Outside of windows, pyenv is pretty decent at managing python versions and virtual environments. I wish there was some equivalent tool for windows. Maybe pyenv for windows will gain some traction?

My recommendation is that:

• we support the management of pip-only dependencies in a conda virtual environment (since this seems
• we explicitly add to the docs somewhere that we don't support
  • installation of conda packages
  • publishing of conda packages
  • managing of conda virtual environments

[underchemist]

In terms of poetry installing packages to the base conda environment despite being in a conda virtualenv, I have been able to avoid this somewhat by installing poetry from get-poetry.py instead of pip from within my conda base environment. My poetry config settings are like those above with settings.virtualenvs.create false and settings.virtualenvs.path <miniconda/env/path>.

[finswimmer]

This issue should be solved by PR #1432 , shouldn't it?

[kasteph]

@finswimmer yes, it should be. That PR will be merged shortly so I will close this issue now.

[nlhnt]

I don't think the PR fixed the poetry show -v command. The below text is copied from my comment in #1432

I don't think this is working, speaking from Windows. Same thing as above happesn for pytest and anything else I'm trying to use in a poetry / conda project.

poetry show -v with dependencies commented out:

$ poetry show -v
$ Using virtualenv: C:\Users\username\.conda\envs\phy_sec

poetry show -v with dependencies defined, numpy = "^1.17"

Using virtualenv: C:\Users\username\.conda\envs\phy_sec

  Stack trace:

  6  ~\.conda\envs\phy_sec\lib\site-packages\clikit\console_application.py:131 in run
     status_code = command.handle(parsed_args, io)

  5  ~\.conda\envs\phy_sec\lib\site-packages\clikit\api\command\command.py:120 in handle
     status_code = self._do_handle(args, io)

  4  ~\.conda\envs\phy_sec\lib\site-packages\clikit\api\command\command.py:171 in _do_handle
     return getattr(handler, handler_method)(args, io, self)

  3  ~\.conda\envs\phy_sec\lib\site-packages\cleo\commands\command.py:92 in wrap_handle
     return self.handle()

  2  ~\.conda\envs\phy_sec\lib\site-packages\poetry\console\commands\show.py:82 in handle
     ops = solver.solve()

  1  ~\.conda\envs\phy_sec\lib\site-packages\poetry\puzzle\solver.py:65 in solve
     packages, depths = self._solve(use_latest=use_latest)

  SolverProblemError

  Because pyforming depends on numpy (^1.17) which doesn't match any versions, version solving failed.

  at ~\.conda\envs\phy_sec\lib\site-packages\poetry\puzzle\solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

conda list with grep to look for a dependency installed in conda env under test

conda list | select-string "numpy"
numpy                     1.17.4                   pypi_0    pypi
numpydoc                  0.9.1                      py_0    conda-forge

[adamgranthendry]

@sdispater I think everyone is missing a key point here: poetry only support pure python wheels. conda comes with many packages built and optimized for speed targeting specific platforms, which is needed for many scientific applications (e.g. numpy+mkl, Intel embree, support for SSE and AVX instruction sets, etc.). Until poetry can support these packages for heavy-lifting data science applications, I don't think poetry will be much use at all to the data scientist.

[sgbaird]

@adamgranthendry I was using flit to publish to PyPI, followed by using grayskull to create a conda recipe and then upload to Anaconda. I've been able to use it for various "data-science" packages:

dependencies = [
    "tqdm",
    "pqdm", #PyPI
    "seaborn",
    "pymatgen",
    "scipy",
    "hdbscan",
    "cython", # for hdbscan 0.8.27 or similar
    "numba>=0.53.1",
    "plotly",
    "python-kaleido",
    "pandas",
    "matplotlib>=3.4.3",
    "joblib",
    "ElMD==0.4.8", #PyPI
    "psutil",
    "scikit_learn",
    "umap-learn",
    "dill",
]

Just now trying out poetry instead.

[xiaoxiangmoe]

https://pixi.sh/ is a package manager that is very similar to poetry, but supports conda natively.
If what you need is venv + lockfile + conda, this tool should be suitable.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.