Locking dependencies fails when using python version constraints

[thueringa]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian bookworm, Python 3.9.12, Python 3.7.13
• Poetry version: 1.2.0b2.dev0
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/thueringa/a9fe9ea20408a5d1e54af4a8bd1eda58

Issue

I have the following dependencies:

[tool.poetry.dependencies]
python = "^3.7"
flake8 = "4.0.1"
sphinx = [
    {version = "4.3.1", python = "~3.7.0"},
    {version = "^4.5.0", python = "^3.8"},
]

Locking the depencies with poetry lock --no-update fails with the following error (see Gist for full output of poetry lock -vvv --no-update):

  SolverProblemError

  Because flake8 (4.0.1) depends on importlib-metadata (<4.3)
   and sphinx (4.5.0) depends on importlib-metadata (>=4.4), flake8 (4.0.1) is incompatible with sphinx (4.5.0).
  And because no versions of sphinx match >4.5.0,<5.0.0, flake8 (4.0.1) is incompatible with sphinx (>=4.5.0,<5.0.0).
  So, because poetry-test depends on both flake8 (4.0.1) and sphinx (^4.5.0), version solving failed.

  at ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:155 in _solve
      151│             packages = result.packages
      152│         except OverrideNeeded as e:
      153│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      154│         except SolveFailure as e:
    → 155│             raise SolverProblemError(e)
      156│ 
      157│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      158│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      159│ 

I would expect poetry to find a solution where flake8==4.0.1 is installed for all environments, sphinx==4.3.1 is installed for Python 3.7.* environments, and sphinx==4.5.0 is installed for all Python 3.8 environments and upwards.

A work-around is to define specific versions of flake8 for each Python environment:

[tool.poetry.dependencies]
python = "^3.7"
flake8 = [
    {version = "4.0.0", python = "~3.7.0"},
    {version = "4.0.1", python = "^3.8"},
]
sphinx = [
    {version = "4.3.1", python = "~3.7.0"},
    {version = "^4.5.0", python = "^3.8"},
]

This correctly locks the dependencies, but it only works when using different versions of flake8 for each kind of environment, which is impractical.

Additional information

Similar failing command

A similar set of dependencies that fails is this one:

[tool.poetry.dependencies]
python = "^3.7"
importlib-metadata = {version ="<4.3", python= "~3.7.0"}
sphinx = [
    {version = "4.3.1", python = "~3.7.0"},
    {version = "^4.5.0", python = "^3.8"},
]

I would expect poetry to find a solution for Python 3.8 environments and upwards that installs sphinx==4.5.0 and a compatible version of importlib-metadata that is defined in the dependencies of sphinx v4.5.0. For Python 3.7 environments I would expect poetry to find a solution that installs sphinx==4.3.1 and a version of importlib-metadata that adheres to our dependency constraint.

importlib-metadata dependencies in referenced packages flake8 and sphinx

The relevant lines concerning importlib-metadata in setup.cfg of flake8 v4.0.1 (unchanged from flake8 v4.0.0):

install_requires =
    [...]
    importlib-metadata<4.3;python_version<"3.8"

In setup.py of sphinx v4.5.0:

install_requires = [
    [...]
    "importlib-metadata>=4.4; python_version < '3.10'",
]

sphinx v4.3.1 does not seem to have a dependency on importlib-metadata.

Possibly related issues: #5506, #5447, #4695

[Qu4tro]

Hey there. This worked for me:

# pyproject.toml (lines 61-64)
sphinx = [
    {version = "^4.5.0", python = ">=3.10"},
    {version = "~4.3.0", python = "<3.10"},
]

I have another problem though, after that. Having sphinx-rtd-theme = "^1.0.0" and trying to do poetry export and pip install, results in:

ERROR: Cannot install sphinx==4.3.2 and sphinx==4.5.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested sphinx==4.3.2
    The user requested sphinx==4.5.0

Seems to be somewhat similar to #5407.

I "fixed" that, by pinning sphinx to "~4.3.0".

[thueringa]

Hi, thanks for your reply. Indeed, locking dependencies seems to work with your snippet (at least for sphinx and flake8, not sure about your issue with sphinx-rtd-theme). What makes it work is that the sphinx version split is at Python 3.10 instead of Python 3.8. But I don't understand why it fails with the split at Python 3.8, as sphinx v4.5.0 is of course compatible with Python 3.8 and 3.9 and does only have a dependency on importlib-metadata for Python < 3.8 according to its setup.py.

Looking at the issue you referenced, this might be some kind of inconsistency in the way sphinx define their dependencies in their source repository, versus how they are defined in the pypi package information? I am not sure how one could verify that.

[thueringa]

I have tested this out with different versions of poetry:

Using this snippet:

[tool.poetry.dependencies]
python = "^3.7"
flake8="4.0.1"
sphinx = [
    {version = "^4.5.0", python = ">=3.8"},
    {version = "4.3.1", python = "<3.8"},
]

With Poetry v1.0.10 dependency locking works, and with Poetry v1.1.0 dependency locking fails with the same error message as in my opening post.

Using git bisect, I found 134aef1 to be the first commit where dependency locking fails.

[Qu4tro]

Small nitpick:

sphinx v4.5.0 is of course compatible with Python 3.8 and 3.9 and does only have a dependency on importlib-metadata for Python < 3.8 according to its setup.py.

I think you mean Flake8, when you say "have a dependency on importlib-metadata for Python < 3.8 according to its setup.py."
As you've mentioned in the initial post, on sphinx you have importlib-metadata>=4.4; python_version < '3.10', but indeed on flake8 you have importlib-metadata<4.3;python_version<"3.8". Right?

Nevertheless, this indeed seems like an issue with Poetry.

[thueringa]

Sorry, yes, I got all the versions a bit mixed up in my head it seems...
To clarify:

• flake8 depends on importlib-metadata<4.3;python_version<"3.8",
• sphinx v4.5.0 depends on "importlib-metadata>=4.4; python_version < '3.10'",
• sphinx v4.3.* has no dependency on importlib-metadata

So yes, I believe this is a regression bug of Poetry, as it should find a solution for this set of dependencies with the snippet above, like it did in v1.0.10.

[thueringa]

Some more info that might help narrow this down:

Using this snippet, which lets Poetry (version 1.1.13) do the dependency resolving around Sphinx:

[tool.poetry.dependencies]
python = "^3.7"
flake8="4.0.1"
sphinx = ">=4.3"

Poetry will lock dependencies successfully, but only install sphinx==4.3.2 regardless of env version, even though sphinx==4.5.0 would be viable for 3.8 upwards.
In the output of poetry lock -vvv it says:

   1: fact: flake8 (4.0.1) depends on importlib-metadata (<4.3)

Restricting the supported python versions with python = "^3.8" or higher will correctly install sphinx==4.5.0, but then 3.7 envs will no longer be supported by the project.

It seems that Poetry incorrectly applies the dependency on importlib-metadata of Flake8 to all python envs that are supported in pyproject.toml, instead of just to 3.8 and below. On the other hand, when environments below 3.8 are not supported at all in pyproject.toml, this dependency is ignored completely by the dependency solver, which is correct.

[thueringa]

I should have thought about the set of dependencies diligently, I see now that this was an misunderstanding on my side on the dependencies of Sphinx v4.5.0. It does have a dependency on "importlib-metadata>=4.4; python_version < '3.10'". So the behavior of Poetry in conjunction with Flake8 is correct, and the snippet in my opening post is not expected to be solvable.

Thank you for your help @Qu4tro, I will close this issue now.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.