Poetry add and init can create incompatible settings in pyproject.toml

[nottrobin]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu Xenial
• Poetry version: 0.12.10
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/nottrobin/0cb0ab3a049891640280fa08f4ca5bcc

Issue

If I try to create a pyproject.toml with Python2 but with Django as a dependency, it creates a pyproject.toml that is incompatible:

$ poetry init --no-interaction --dependency django

This command will guide you through creating your pyproject.toml config.


Using version ^2.1 for django

$ poetry install
Creating virtualenv django-project-py2.7 in /root/.cache/pypoetry/virtualenvs
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

[SolverProblemError]
The current project must support the following Python versions: 2.7.12        
Because no versions of django match >=2.1,<2.1.4 || >2.1.4,<3.0               
 and django (2.1.4) requires Python >=3.5, django is forbidden.               
So, because django-project depends on django (^2.1), version solving failed.  

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

This is because these two requirements are fundamentally not quite compatible:

python = "^2.7"
django = "^2.1"

I understand that this might be difficult for Poetry to resolve as it might not have the information that Django 2.1 doesn't work on python < 3.5 at the time it's creating the pyproject.toml. But it's unfortunate that this means that the setup experience for a poetry-based Django project requires such deeper understanding.

Possible solutions could be:

1. poetry init --dependency django would set the version of python in pyproject.toml to ^3.5 (maybe with a warning that your current version of python is lower than that)
2. poetry init --dependency django would error if the current python version was unsupported
3. poetry init --dependency django would select the most recent version of Django supported by the current version of python

I think the nicest experience would be 1. - as you might want to generate a valid pyproject.toml with any version of python, in the full knowledge that you can't actually run the project with that version.

[sdispater]

Poetry should select the proper constraint for Django by using the Python constraint of the project. So in this case it should use "^1.8" (So your point 3.).

Point 1 would seem odd since you would get an error later on saying that the current project is not compatible with the currently activated Python version. So 3 would provide the most intuitive experience.

[nottrobin]

@sdispater thanks for responding so quickly. Yes I see your point.

I was thinking that maybe you consider the project itself (as defined by pyproject.toml) as a separate entity from the person running poetry init - so the project should be setup in the most modern way (therefore, Django 2.1 and Python ^3.5), and then only when you do poetry install would it say you need Python 3.5.

But I agree that would be pretty frustrating for most normal cases, where you'd definitely want to set up the project and then immediately start using it. So, yes, 3. would be the ideal.

[danielniccoli]

I understand that this might be difficult for Poetry to resolve as it might not have the information that Django 2.1 doesn't work on python < 3.5 at the time it's creating the pyproject.toml.

I don't think that the problem. What I see is that poetry will always resolve to the latest version and try to install it.

Take this example:

PS C:\Users\Daniel\Desktop\myproject> poetry init --python=~3.7 -n -v
PS C:\Users\Daniel\Desktop\myproject> poetry env use C:\Users\Daniel\AppData\Local\Programs\Python\Python37\python.exe
Creating virtualenv myproject in C:\Users\Daniel\Desktop\myproject\.venv
Using virtualenv: C:\Users\Daniel\Desktop\myproject\.venv
PS C:\Users\Daniel\Desktop\myproject> poetry install
Updating dependencies
Resolving dependencies...

Writing lock file
PS C:\Users\Daniel\Desktop\myproject> poetry lock
Updating dependencies
Resolving dependencies...
PS C:\Users\Daniel\Desktop\myproject> poetry add django
Using version ^4.1.3 for django

Updating dependencies
Resolving dependencies...

The current project's Python requirement (>=3.7,<3.8) is not compatible with some of the required packages Python requirement:
  - django requires Python >=3.8, so it will not be satisfied for Python >=3.7,<3.8

Because no versions of django match >4.1.3,<5.0.0
 and django (4.1.3) requires Python >=3.8, django is forbidden.
So, because myproject depends on django (^4.1.3), version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties

    For django, a possible solution would be to set the `python` property to "<empty>"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers
PS C:\Users\Daniel\Desktop\myproject>

I create a new repository and set the python version to 3.7.*.

Next, I add django as a dependency. Django dropped support for Python 3.7 with Django 4.0 (docs). Poetry defaults to resolve to the latest available version.

It then tries to install django in that version, but realises that it is not compatible to version 3.7.

And instead of finding a compatible version, which would be <4.0, it just gives up. It even suggests to remove the Python version marker, which is the exact opposite of what I want to do.

This behaviour is not helpful. Instead of supporting me in my endeavour, I'm expected to adapt to the tool. But then I ask myself, what do I need poetry for, when it just adds to tool-bloat, when I could just use pip with the same effort.

Instead of giving up, poetry should get the highest compatible version to the previous dependency (which is python 3.7.*) and install that. This does also not only apply to Django, but to basically every package you add to a project.

[rphes]

Completely agree with #707 (comment) above. Yesterday I found this makes working with older versions of Python a pain. Is this behavior intended or are we running into a bug here?

[caseytomlin]

I guess I have some fundamental misunderstanding of intended usage. The main docs page says

Poetry comes with an exhaustive dependency resolver, which will always find a solution if it exists.

What is the recommended way to e.g., use the most recent set of compatible versions for all dependencies? My failure and workaround is probably not hitting on those most recent versions.

[radoering]

Your issue is not the resolver but the semantics of poetry add:
IIRC poetry add foo is the same as poetry add foo@latest. What you expect is probably poetry add foo@* (or something more sophisticated).

[mkp-bangemann]

The docs for poetry add say the following:

If you do not specify a version constraint, poetry will choose a suitable one based on the available package versions.

Which later is followed up by

If you want to get the latest version of an already present dependency, you can use the special latest constraint: poetry add pendulum@latest

So I would expect poetry add to behave along the lines of poetry add foo@* as it's trying to find a "suitable version".

Regarding my current use case - I am trying to install the package numpy for python 3.8.* using poetry. The newest version of numpy requires python3.9, so poetry add numpy fails with the initial error of this issue. I can still install the version i need using poetry add numpy@*. Thank you for that #707 (comment)!

Something to note is that Pycharm for example (haven't tested vscode yet), also uses poetry add foo, so using the UI of the IDE to install packages for older python versions fails. Now you could say that Pycharm is using the wrong command or I just expect a different behaviour, but I would expect poetry add numpy to behave like poetry add numpy@* instead of of poetry add numpy@latest, especially when considering the documentation.

Edit
I don't suggest that the behaviour of the commands should be mixed up, since that would obviously have it's own problems. But I have seen at least four other Issues where this misconception was the root of the problem.
So I believe it would be useful to add a note to the docs clarifying this behaviour, since this is the first place I go to when being confronted with unexpected behaviour.

[dimbleby]

docs improvements are a great way to contribute, please go ahead

[Alexander-Serov]

Hello,

Just to confirm after reading all the comments: is there a roadmap for solving this issue? It seems like the comments above clearly define the issue.

My personal two cents:
If the latest version of django is 4.1.0 and I type poetry add django, as mentioned above, it interprets it as poetry add django^4.1.0 and then fails for incompatibility. What I end up doing myself as a hotfix is poetry add django<4.1.0 and sometimes iterating until I find the compatible one. It doesn't put the right requirements into pyproject.toml but most times allows me to advance with my task. Perhaps something similar could be implemented in the interpretation of the command by poetry? For instance, once the compatible version X is found just update the pyproject.toml with `django = "^X"?

[SergeyP1990]

Just to mention - with pip you just do install and it installs latest version of package for current version of python