Cannot install numpy and opencv-python with specified versions that should work together due to wrongly detected numpy requirement

[p-wojciechowski]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 20.04.5 LTS (Intel), Python 3.9.7
• Poetry version: 1.2.0
• Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

In my project I need to install opencv-python==4.5.5.62 and numpy==1.19.5 with these specified versions. This problem is similar to #4451 but this time JSON metadata about opencv-python requirements have proper values:

curl -s https://pypi.org/pypi/opencv-python/4.5.5.62/json | jq '.info.requires_dist'
[
  "numpy (>=1.13.3) ; python_version < \"3.7\"",
  "numpy (>=1.21.2) ; python_version >= \"3.10\"",
  "numpy (>=1.21.2) ; python_version >= \"3.6\" and platform_system == \"Darwin\" and platform_machine == \"arm64\"",
  "numpy (>=1.19.3) ; python_version >= \"3.6\" and platform_system == \"Linux\" and platform_machine == \"aarch64\"",
  "numpy (>=1.14.5) ; python_version >= \"3.7\"",
  "numpy (>=1.17.3) ; python_version >= \"3.8\"",
  "numpy (>=1.19.3) ; python_version >= \"3.9\""
]

Which is properly read when installing with pip (see Collecting numpy>=1.19.3):

pip3 install opencv-python==4.5.5.62
Collecting opencv-python==4.5.5.62
  Using cached opencv_python-4.5.5.62-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.4 MB)
Collecting numpy>=1.19.3
  Downloading numpy-1.23.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.1/17.1 MB 43.2 MB/s eta 0:00:00
Installing collected packages: numpy, opencv-python
Successfully installed numpy-1.23.2 opencv-python-4.5.5.62

opencv-python==4.5.5.62 and numpy==1.19.5 can be installed together with pip, however it can't be installed with poetry.

After adding opencv-python==4.5.5.62 to project, numpy==1.19.5 can't be added because poetry reads wrongly numpy version requirement as >=1.21.2

$ poetry add numpy==1.19.5 -vvv
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Loading configuration file /home/pawel/.config/pypoetry/config.toml
Using virtualenv: /home/pawel/.pyenv/versions/3.9.7/envs/testing
Project environment contains an empty path in sys_path, ignoring.
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
No suitable keyring backend found
No suitable keyring backends were found
Keyring is not available, credentials will be stored and retrieved from configuration files as plaintext.
Creating new session for pypi.org
Source (PyPI): No release information found for numpy-0.9.6, skipping
Source (PyPI): No release information found for numpy-0.9.8, skipping
Source (PyPI): No release information found for numpy-1.0.3, skipping
Source (PyPI): No release information found for numpy-1.0.4, skipping
Source (PyPI): No release information found for numpy-1.0b1, skipping
Source (PyPI): No release information found for numpy-1.0b4, skipping
Source (PyPI): No release information found for numpy-1.0b5, skipping
Source (PyPI): No release information found for numpy-1.0rc1, skipping
Source (PyPI): No release information found for numpy-1.0rc2, skipping
Source (PyPI): No release information found for numpy-1.0rc3, skipping
Source (PyPI): No release information found for numpy-1.1.1, skipping
Source (PyPI): No release information found for numpy-1.2.0, skipping
Source (PyPI): No release information found for numpy-1.2.1, skipping
Source (PyPI): No release information found for numpy-1.4.0, skipping
Source (PyPI): 1 packages found for numpy 1.19.5

Updating dependencies
Resolving dependencies...
   1: fact: example is 0.1.0
   1: derived: example
   1: fact: example depends on opencv-python (4.5.5.62)
   1: fact: example depends on numpy (1.19.5)
   1: selecting example (0.1.0)
   1: derived: numpy (==1.19.5)
   1: derived: opencv-python (==4.5.5.62)
Source (PyPI): No release information found for numpy-0.9.6, skipping
Source (PyPI): No release information found for numpy-0.9.8, skipping
Source (PyPI): No release information found for numpy-1.0.3, skipping
Source (PyPI): No release information found for numpy-1.0.4, skipping
Source (PyPI): No release information found for numpy-1.0b1, skipping
Source (PyPI): No release information found for numpy-1.0b4, skipping
Source (PyPI): No release information found for numpy-1.0b5, skipping
Source (PyPI): No release information found for numpy-1.0rc1, skipping
Source (PyPI): No release information found for numpy-1.0rc2, skipping
Source (PyPI): No release information found for numpy-1.0rc3, skipping
Source (PyPI): No release information found for numpy-1.1.1, skipping
Source (PyPI): No release information found for numpy-1.2.0, skipping
Source (PyPI): No release information found for numpy-1.2.1, skipping
Source (PyPI): No release information found for numpy-1.4.0, skipping
Source (PyPI): 1 packages found for numpy 1.19.5
   1: selecting numpy (1.19.5)
   0: Duplicate dependencies for numpy
   0: Different requirements found for numpy (>=1.21.2) with markers python_version >= "3.10" or python_version >= "3.6" and platform_system == "Darwin" and platform_machine == "arm64", numpy (>=1.19.3) with markers python_version >= "3.6" and platform_system == "Linux" and platform_machine == "aarch64" or python_version >= "3.9", numpy (>=1.14.5) with markers python_version >= "3.7" and numpy (>=1.17.3) with markers python_version >= "3.8".
   1: Version solving took 0.056 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}}).
   1: fact: example is 0.1.0
   1: derived: example
   1: fact: example depends on opencv-python (4.5.5.62)
   1: fact: example depends on numpy (1.19.5)
   1: selecting example (0.1.0)
   1: derived: numpy (==1.19.5)
   1: derived: opencv-python (==4.5.5.62)
Source (PyPI): No release information found for numpy-0.9.6, skipping
Source (PyPI): No release information found for numpy-0.9.8, skipping
Source (PyPI): No release information found for numpy-1.0.3, skipping
Source (PyPI): No release information found for numpy-1.0.4, skipping
Source (PyPI): No release information found for numpy-1.0b1, skipping
Source (PyPI): No release information found for numpy-1.0b4, skipping
Source (PyPI): No release information found for numpy-1.0b5, skipping
Source (PyPI): No release information found for numpy-1.0rc1, skipping
Source (PyPI): No release information found for numpy-1.0rc2, skipping
Source (PyPI): No release information found for numpy-1.0rc3, skipping
Source (PyPI): No release information found for numpy-1.1.1, skipping
Source (PyPI): No release information found for numpy-1.2.0, skipping
Source (PyPI): No release information found for numpy-1.2.1, skipping
Source (PyPI): No release information found for numpy-1.4.0, skipping
Source (PyPI): 1 packages found for numpy 1.19.5
   1: selecting numpy (1.19.5)
   1: fact: opencv-python (4.5.5.62) depends on numpy (>=1.21.2)
   1: conflict: opencv-python (4.5.5.62) depends on numpy (>=1.21.2)
   1: ! opencv-python (==4.5.5.62) is satisfied by opencv-python (==4.5.5.62)
   1: ! which is caused by "example depends on opencv-python (4.5.5.62)"
   1: ! thus: numpy is required
   1: ! not numpy (>=1.21.2) is satisfied by numpy (==1.19.5)
   1: ! which is caused by "example depends on numpy (1.19.5)"
   1: ! thus: version solving failed
   1: Version solving took 0.025 seconds.
   1: Tried 1 solutions.

  Stack trace:

  4  ~/.local/lib/python3.8/site-packages/poetry/puzzle/solver.py:151 in _solve
      149│
      150│         try:
    → 151│             result = resolve_version(
      152│                 self._package, self._provider, locked=locked, use_latest=use_latest
      153│             )

  3  ~/.local/lib/python3.8/site-packages/poetry/mixology/__init__.py:24 in resolve_version
       22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       23│
    →  24│     return solver.solve()
       25│

  2  ~/.local/lib/python3.8/site-packages/poetry/mixology/version_solver.py:127 in solve
      125│             while next is not None:
      126│                 self._propagate(next)
    → 127│                 next = self._choose_package_version()
      128│
      129│             return self._result()

  1  ~/.local/lib/python3.8/site-packages/poetry/mixology/version_solver.py:446 in _choose_package_version
      444│             package = locked
      445│
    → 446│         package = self._provider.complete_package(package)
      447│
      448│         conflict = False

  OverrideNeeded

  ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}}, {Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}}, {Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}}, {Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}})

  at ~/.local/lib/python3.8/site-packages/poetry/puzzle/provider.py:768 in complete_package
      764│                     current_overrides.update({dependency_package: package_overrides})
      765│                     overrides.append(current_overrides)
      766│
      767│             if overrides:
    → 768│                 raise OverrideNeeded(*overrides)
      769│
      770│         # Modifying dependencies as needed
      771│         clean_dependencies = []
      772│         for dep in dependencies:

The following error occurred when trying to handle this error:


  Stack trace:

  4  ~/.local/lib/python3.8/site-packages/poetry/puzzle/solver.py:151 in _solve
      149│
      150│         try:
    → 151│             result = resolve_version(
      152│                 self._package, self._provider, locked=locked, use_latest=use_latest
      153│             )

  3  ~/.local/lib/python3.8/site-packages/poetry/mixology/__init__.py:24 in resolve_version
       22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       23│
    →  24│     return solver.solve()
       25│

  2  ~/.local/lib/python3.8/site-packages/poetry/mixology/version_solver.py:126 in solve
      124│             next: str | None = self._root.name
      125│             while next is not None:
    → 126│                 self._propagate(next)
      127│                 next = self._choose_package_version()
      128│

  1  ~/.local/lib/python3.8/site-packages/poetry/mixology/version_solver.py:165 in _propagate
      163│                     # where that incompatibility will allow us to derive new assignments
      164│                     # that avoid the conflict.
    → 165│                     root_cause = self._resolve_conflict(incompatibility)
      166│
      167│                     # Back jumping erases all the assignments we did at the previous

  SolveFailure

  Because example depends on opencv-python (4.5.5.62) which depends on numpy (>=1.21.2), numpy is required.
  So, because example depends on numpy (1.19.5), version solving failed.

  at ~/.local/lib/python3.8/site-packages/poetry/mixology/version_solver.py:364 in _resolve_conflict
      360│             )
      361│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      362│             self._log(f"! thus: {incompatibility}")
      363│
    → 364│         raise SolveFailure(incompatibility)
      365│
      366│     def _choose_package_version(self) -> str | None:
      367│         """
      368│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  Stack trace:

  13  ~/.local/lib/python3.8/site-packages/cleo/application.py:329 in run
       327│
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  12  ~/.local/lib/python3.8/site-packages/poetry/console/application.py:185 in _run
       183│         self._load_plugins(io)
       184│
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│

  11  ~/.local/lib/python3.8/site-packages/cleo/application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│

  10  ~/.local/lib/python3.8/site-packages/cleo/application.py:465 in _run_command
       463│
       464│         if error is not None:
     → 465│             raise error
       466│
       467│         return event.exit_code

   9  ~/.local/lib/python3.8/site-packages/cleo/application.py:449 in _run_command
       447│
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   8  ~/.local/lib/python3.8/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│
     → 119│         status_code = self.execute(io)
       120│
       121│         if status_code is None:

   7  ~/.local/lib/python3.8/site-packages/cleo/commands/command.py:83 in execute
        81│
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

   6  ~/.local/lib/python3.8/site-packages/poetry/console/commands/add.py:251 in handle
       249│         self.installer.whitelist([r["name"] for r in requirements])
       250│
     → 251│         status = self.installer.run()
       252│
       253│         if status == 0 and not self.option("dry-run"):

   5  ~/.local/lib/python3.8/site-packages/poetry/installation/installer.py:111 in run
       109│             self._execute_operations = False
       110│
     → 111│         return self._do_install()
       112│
       113│     def dry_run(self, dry_run: bool = True) -> Installer:

   4  ~/.local/lib/python3.8/site-packages/poetry/installation/installer.py:244 in _do_install
       242│                 source_root=self._env.path.joinpath("src")
       243│             ):
     → 244│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       245│         else:
       246│             self._io.write_line("Installing dependencies from lock file")

   3  ~/.local/lib/python3.8/site-packages/poetry/puzzle/solver.py:73 in solve
        71│         with self._provider.progress():
        72│             start = time.time()
     →  73│             packages, depths = self._solve(use_latest=use_latest)
        74│             end = time.time()
        75│

   2  ~/.local/lib/python3.8/site-packages/poetry/puzzle/solver.py:157 in _solve
       155│             packages = result.packages
       156│         except OverrideNeeded as e:
     → 157│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
       158│         except SolveFailure as e:
       159│             raise SolverProblemError(e)

   1  ~/.local/lib/python3.8/site-packages/poetry/puzzle/solver.py:116 in solve_in_compatibility_mode
       114│             )
       115│             self._provider.set_overrides(override)
     → 116│             _packages, _depths = self._solve(use_latest=use_latest)
       117│             for index, package in enumerate(_packages):
       118│                 if package not in packages:

  SolverProblemError

  Because example depends on opencv-python (4.5.5.62) which depends on numpy (>=1.21.2), numpy is required.
  So, because example depends on numpy (1.19.5), version solving failed.

  at ~/.local/lib/python3.8/site-packages/poetry/puzzle/solver.py:159 in _solve
      155│             packages = result.packages
      156│         except OverrideNeeded as e:
      157│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      158│         except SolveFailure as e:
    → 159│             raise SolverProblemError(e)
      160│
      161│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      162│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      163│

[clintonroy]

I think, pip is taking into account the local system and architecture, where as poetry is trying to create a universal installation solution, so the darwin/arm64 lines are being taken into account. I do not think there's any work around for that at the moment, could be wrong though.

[p-wojciechowski]

Poetry's documentation shows that there is a possibility to specify install conditions for dependencies based on environment markers.

[dimbleby]

Yes - suggest that you try that!

ie add an explicit dependency for the numpy that you want, in the environment that you want it

[p-wojciechowski]

@dimbleby but it is the problem with reading opencv-python requirements by poetry, not requirements of my exemplary project. After adding these lines to pyproject.toml the problem is still there.

[tool.poetry.dependencies]
python = "^3.9"
opencv-python= {version="4.5.5.62", markers="python_version>= \"3.9\""}
numpy= {version="1.19.5", markers="python_version>= \"3.9\""}

In -vvv verbose logs that I posted before, there is a line where poetry sees conditional numpy requirements, but after 2 lines it decides to override them:

0: Different requirements found for numpy (>=1.21.2) with markers python_version >= "3.10" or python_version >= "3.6" and platform_system == "Darwin" and platform_machine == "arm64", numpy (>=1.19.3) with markers python_version >= "3.6" and platform_system == "Linux" and platform_machine == "aarch64" or python_version >= "3.9", numpy (>=1.14.5) with markers python_version >= "3.7" and numpy (>=1.17.3) with markers python_version >= "3.8".
1: Version solving took 0.056 seconds.
1: Tried 1 solutions.
0: Retrying dependency resolution with the following overrides ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}})

[dimbleby]

poetry is trying to produce a solution that will work on all platforms, for all versions of python supported by your pyproject.toml.

If you unconditionally require numpy=1.19.5 then there is no such solution, because on some platforms this conflicts with the requirements of opencv-python

The same is true in your most recent try, that requirement is in conflict with this opencv-python requirement

"numpy (>=1.21.2) ; python_version >= \"3.6\" and platform_system == \"Darwin\" and platform_machine == \"arm64\"",

You will have to construct a requirement that specifies the platform that you are on, and that does not cause conflicts with opencv-python requirements

[p-wojciechowski]

I set requirements with exact platform info, but unfortunately it didn't help:

opencv-python= {version="4.5.5.62", markers="platform_machine == 'x86_64' and sys_platform == 'Linux' and python_version >= '3.9'"}
numpy= {version="1.19.5", markers="platform_machine == 'x86_64' and sys_platform == 'Linux' and python_version >= '3.9'"}  

Poetry still enforce numpy >= 1.21.2.

[dimbleby]

this looks like it should work, failure is probably duplicate #5506

[dimbleby]

actually no, I take it back, you're still conflicting this opencv-python requirement

"numpy (>=1.21.2) ; python_version >= \"3.10\"",

[p-wojciechowski]

opencv-python= {version="4.5.5.62", markers="python_version < '3.10'"}
numpy= {version="1.19.5", markers="python_version < '3.10'"}  

fixed the problem. Thank You for your suggestions!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.