Poetry installing dev/beta releases

[nayyarv]

I think this may be more to do with the way certain packages release on pip, in my case pytorch and tensorflow - I've not noticed this behaviour in other packages to date.
Combined with neural nets being used to generate poetry, it's tricky to google search for this so apologies if this is a duplicate.

Ubuntu 18.04
Poetry 0.11.5

Tensorflow toml

[tool.poetry.dependencies]
python = "^3.6"
numpy = "^1.15.0"
scipy = "^1.1.0"
scikit-learn = "^0.19.2"
tensorflow = "^1.10.1"
tensorflow-gpu = "^1.10.1"

  - Installing scikit-learn (0.19.2)
  - Installing tensorflow (1.11.0rc0)
  - Installing tensorflow-gpu (1.11.0rc0)
  - Installing ujson (1.35)

In this case this happens because I've specified numpy 1.15, only the 1.11 rc is able to work correctly. I'm not sure this is optimal behaviour - I shouldn't get an alpha release in this case. Once I bump numpy to 1.14 I get 1.10.1 again.

In pytorch, it's a bit different and I'm not sure what happens

[tool.poetry.dependencies]
python = "^3.6"
numpy = "^1.15.0"
scipy = "^1.1.0"
scikit-learn = "^0.19.2"
torch ="^0.4.1"
torchvision = "^0.2.1"

➜ poetry update
Updating dependencies
Resolving dependencies... (48.5s)

Package operations: 7 installs, 0 updates, 0 removals

Writing lock file
...
  - Installing pillow (5.2.0)
  - Installing torch (0.4.1.post2)
                                                                                                                                         
[VenvCommandError]                                                                                                        
Command ['pip', 'install', '--no-deps', 'torch==0.4.1.post2'] errored with the following output:                                       
Collecting torch==0.4.1.post2

Again doing pip3 install --user torch torchvision will install 0.4.1 while poetry tries to install the post2 release. I'm not sure what's going on in this version tbh. I've worked around by setting equal for the time being.

[nayyarv]

To add some colour to the torch section above, 0.4.1.post2 is the latest version, but only for python3.7. Since I'm running python 3.6, i should get the 0.4.1 release in this case.

import collections
import requests

torch = requests.get("https://pypi.org/pypi/torch/json")
dat = torch.json()

supported = collections.defaultdict(set)
for releaseVal, release in dat["releases"].items():
    for subver in release:
        supported[releaseVal].add(subver['python_version'])
            
dict(supported)

{'0.1.2': {'source'},
 '0.1.2.post1': {'source'},
 '0.3.0.post4': {'cp27', 'cp35', 'cp36'},
 '0.3.1': {'cp27', 'cp35', 'cp36'},
 '0.4.0': {'cp27', 'cp35', 'cp36'},
 '0.4.1': {'cp27', 'cp35', 'cp36', 'cp37'},
 '0.4.1.post2': {'cp37'}}

Additionally, looking at your pypi code, I don't ever see the python_version field being parsed out. Instead I see

        if release_info["requires_python"]:
            package.python_versions = release_info["requires_python"]

Now, having had a look around, I can see that the use of requires_python is relatively recent and not widely spread and the packages that use the old style should also be supported.

On a sidenote

The created virtualenv will use the Python executable for which poetry has been installed.

seems counterintuitive. If I install it into python3.6 but want to code in python2.7 as per the toml, how would I do that?
Do you provide a poetry2, poetry3 like pip?

Wouldn't having

[tool.poetry.dependencies]
python = "^3.7"

make a 3.7 venv (if possible)
and

[tool.poetry.dependencies]
python = "3.4"

make a 3.4 venv (if possible)
shouldn't that use the python version as specified for the venv?

[nayyarv]

To clarify the above, I don't think I should have to activate the right virtualenv to set up a poetry project.

[sdispater]

It has been discussed before and the decision remains: Poetry is not and never will be a Python management tool. This is something that tools like pyenv do really well and is not something a package/dependency manager should do.

And if you declare python = "~2.7 || ^3.4" which Python version should Poetry use?

Note however that with the next 0.12 release, the recommended installer will only need to be used once. The poetry tool will pick up the currently activated Python version and use it to create the virtualenv.

Basically:

$ curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python
# Once installed you can just switch Python versions with tools like pyenv
$ pyenv install 3.7.0
$ pyenv local 3.7.0
$ poetry debug:info
Poetry
======

 * Version: 0.12.0a2
 * Python:  3.7.0


Virtualenv
==========

 * Python:         3.7.0
 * Implementation: CPython
 * Path:           /path/to/venv-3.7/


System
======

 * Platform: darwin
 * OS:       posix
 * Python:   /usr/local/var/pyenv/versions/3.7.0

$ pyenv local 3.6.5
$ poetry debug:info
Poetry
======

 * Version: 0.12.0a2
 * Python:  3.6.5


Virtualenv
==========

 * Python:         3.6.5
 * Implementation: CPython
 * Path:           /path/to/venv-3.6/


System
======

 * Platform: darwin
 * OS:       posix
 * Python:   /usr/local/var/pyenv/versions/3.6.5

[joshfriend]

The poetry tool will pick up the currently activated Python version and use it to create the virtualenv.

This is something that tools like pyenv do really well and is not something a package/dependency manager should do.

I 1,000% agree with this. Pipenv tries to install python versions using pyenv or pick a matching version of python specified in the `Pipfile, but there are so many caveats and edge cases with trying to figure out the appropriate version of python to use that it almost always gets it wrong in my experience.

Numerous users have asked pyenv to add installation of packages from requirements.txt or Pipfile, and our answer is always "no" for the same reasons.

[nayyarv]

Thanks for the clarification, the syntax of the config files confused my intuition. I think it's fair that poetry doesn't get involved into the pyenv part of things, I guess the lines are blurred a bit since poetry can create the venv if necessary.

However, from my perspective

[tool.poetry.dependencies]
python = "^3.4"
numpy = "^1.0.0"
scipy = "^1.0.0"

will install numpy 1.15, and scipy 1.1.0 but use whatever python version was active when creating the venv. This is counter-intuitive since based on the behaviour of the package installation, I'd expect the .venv to use python3.7. This is inconsistent behaviour imo from the config syntax. I could specify python = 3.8 and it'd be fine too for example.

As a suggestion, separating the python version to a different section or combining with the top section would be better as it'd be instantly more intuitive to me that the python version is not something poetry handles.

Eg., if the config was this

[tool.poetry]
name = "eg"
...
license = "MIT"
python="^3.6"

[tool.poetry.dependencies]
numpy = "^1.0.0"
scipy = "^1.0.0"

I wouldn't have made the same bad assumptions/conclusions re python versioning.

I don't want to get derailed too much, my original concerns still remain:

1. Installing an alpha release of tensorflow to satisfy another dependency is not correct behaviour. Allowing alpha/beta release installation should be explicit.
2. Not parsing python_version in the pypi json means we're not installing the correct version for the project's venv for packages that use that field. As far as I can tell it either uses either source or a list of python versions supported ["cp34", "cp35", "cp36"] or ["3.4", "3.5", "3.6"]

[digitalresistor]

The python version isn't a dependency of poetry though, it is a dependency of the current project.

With

python = "^3.4"

You say that any python version, >=3.4.0,<4.0 will work, so installing into a venv that is version 3.5 is perfectly acceptable, even if your current Python is 3.7. If the venv already exists and was an older version of Python there is no issue with it.

However I think it would be awesome if poetry would recognise that the current active Python (for example using pyenv) does not match the venv, and create a new venv that matches that Python version that is currently active.

Do note that currently if you have Python 3.5 active for poetry, and no venv exists, even with something stating your project depends on python ^3.4 it will still create the venv using Python 3.5, not Python 3.7, so it doesn't try to force the venv to the latest Python version.

[songww]

I think the version of the package that poetry parsed should meet the requirements of the minimum version of Python that is dependent on, rather than satisfying some of the requirements. For example, the project relies on Python>=3.4.0, <4.0, but installing a package is only available for Python 3.7. When my Python version is lower than 3.7, it will fail because it depends on the wrong version. Unless you specify this package to install only under certain python versions.

Otherwise it doesn't make sense to specify a dependent Python version.

Sorry, my English is very poor, this is translated with google.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.