Multiple Python restricted dependencies not solved properly

[Erhanjinn]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 20.04
• Poetry version: 1.1.12
• Link of a Gist with the contents of your pyproject.toml file:

Issue

Hello, I have run into issues with dependency solving using multiple Python restricted dependencies as follows:

...
[tool.poetry.dependencies]
python = "^3.8.6"
py4j = [
    {version = "0.10.9", python = "~3.8"},
    {version = "^0.10.9.2", python = "^3.9"}
]
pyspark = [
    {version = "3.1.2", python = "~3.8"},
    {version = "^3.2.0", python = "^3.9"}
]
...

with the error (after running poetry update):

Updating dependencies
Resolving dependencies... (0.1s)

  SolverProblemError

  Because no versions of pyspark match >3.2.0,<4.0.0
   and pyspark (3.2.0) depends on py4j (0.10.9.2), pyspark (>=3.2.0,<4.0.0) requires py4j (0.10.9.2).
  So, because my-custom-package depends on both py4j (0.10.9) and pyspark (^3.2.0), version solving failed.

  at ~/miniconda3/envs/myenv395/lib/python3.9/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

When I try the different versions for Python ~3.8 and for Python ^3.9 separately as

...
[tool.poetry.dependencies]
python = "^3.8.6"
py4j =  "0.10.9"
pyspark = "3.1.2"
...

and

...
[tool.poetry.dependencies]
python = "^3.8.6"
py4j =  "^0.10.9.2"
pyspark = "^3.2.0"
...

everything goes fine.

Why does poetry mix the Python specific dependencies together and does not respect the Python version conditions?

[Erhanjinn]

Poetry correctly finds the different dependencies

...
0: Different requirements found for py4j (0.10.9) with markers python_version >= "3.8" and python_version < "3.9" and py4j (^0.10.9.2) with markers python_version >= "3.9" and python_version < "4.0".
...
0: Different requirements found for pyspark (3.1.2) with markers python_version >= "3.8" and python_version < "3.9" and pyspark (^3.2.0) with markers python_version >= "3.9" and python_version < "4.0".
...

but then tries to resolve a combination which should not be possible:

Retrying dependency resolution with the following overrides ({Package('my-custom-package', '0.1.0'): {'py4j': <Dependency py4j (==0.10.9)>, 'pyspark': <Dependency pyspark (>=3.2.0,<4.0.0)>}})

[radoering]

Probably, the same issue as in #3367. You may want to check if the changes in the attached PR #4695 solve your issue. (The PR has not yet been reviewed.)

[Erhanjinn]

Thanks for the response @radoering, code from #4695 does however not fix the issue and the error is still the same.

[radoering]

I copied the extract from the pyproject.toml you provided and was able to run poetry lock successfully with the version from the PR. Maybe, there is something more in your pyproject.toml that causes issues? Can you provide a complete sample pyproject.toml so that locking fails using the poetry version from the PR?

Further, due to another issue, I improved #4695 a bit to cover more cases just a moment ago. Maybe, you can check if that coincidentally solves your problem.

[Erhanjinn]

Dear @radoering, I am sorry, I was wrong. Your PR actually fixes the problem and it might have done even before your fix, I think I used a wrong branch of your repo.
Thanks, I confirm the versions get resolved using #4695.

[dschopf]

I have the same problem and verified that the PR does indeed fix it. Are there any plans to merge this in the near future?

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.