Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependency resolution doesn't take platform into account. #224

Closed
4 tasks done
2e0byo opened this issue Aug 2, 2023 · 2 comments
Closed
4 tasks done

Dependency resolution doesn't take platform into account. #224

2e0byo opened this issue Aug 2, 2023 · 2 comments
Labels
duplicate This issue or pull request already exists

Comments

@2e0byo
Copy link

2e0byo commented Aug 2, 2023

  • Poetry version: 1.5.1
  • Python version: 3.10.12
  • OS version and name: (docker: ubuntu, linux 6.4.6)
  • pyproject.toml: So minimal I include it inline:
[tool.poetry]
name = "tmp"
version = "0.1.0"
description = ""
authors = []

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = ">=3.10,<3.11"
torchvision = [
   {platform="linux", url="https://download.pytorch.org/whl/cu118/torchvision-0.15.2%2Bcu118-cp310-cp310-linux_x86_64.whl"},
   {platform="windows", url="https://download.pytorch.org/whl/cu118/torchvision-0.15.2%2Bcu118-cp310-cp310-win_amd64.whl"},
]
sentence-transformers = "^2.2.2"
  • I am on the latest stable Poetry version, installed using a recommended method.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have consulted the FAQ and blog for any relevant entries or release notes.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.
    Attempting to export the above pyproject.toml throws an error, since walker.py thinks there are overlapping

Issue

Attempting to export the above pyproject.toml throws a runtime error. sentence-transformers declares that it depends on torchvision. Currently walker.py doesn't take the platform into account, so it thinks that these dependencies are overlapping and fails. Setting a breakpoint on walker.py::get_locked_package confirms that the explicit passes for torchvision are fine: it's the pass for sentence-transformers which can't work out which to pick.

Looking at the lockfile I can't see where (if anywhere) the platform information is actually stored in it. Should it be? Or should the exporter consult pyproject.toml?

Traceback

(cli-py3.10) app@31552c26b59e ~/c/tmp (ci/folder-vendor)> poetry export -vvv
Loading configuration file /home/app/.config/pypoetry/config.toml
Source (poetry-lockfile): 2 packages found for torchvision *
Source (poetry-lockfile): 2 packages found for torchvision *
Source (poetry-lockfile): 1 packages found for sentence-transformers >=2.2.2,<3.0.0

  Stack trace:

  12  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/cleo/application.py:327 in run
       325│
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  11  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│

  10  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│

   9  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/cleo/application.py:473 in _run_command
       471│
       472│         if error is not None:
     → 473│             raise error
       474│
       475│         return terminate_event.exit_code

   8  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/cleo/application.py:457 in _run_command
       455│
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   7  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│
     → 119│         status_code = self.execute(io)
       120│
       121│         if status_code is None:

   6  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/cleo/commands/command.py:62 in execute
        60│
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

   5  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/poetry_plugin_export/command.py:109 in handle
       107│         exporter.with_credentials(self.option("with-credentials"))
       108│         exporter.with_urls(not self.option("without-urls"))
     → 109│         exporter.export(fmt, Path.cwd(), output or self.io)
       110│
       111│         return 0

   4  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/poetry_plugin_export/exporter.py:80 in export
        78│             raise ValueError(f"Invalid export format: {fmt}")
        79│
     →  80│         getattr(self, self.EXPORT_METHODS[fmt])(cwd, output)
        81│
        82│     def _export_generic_txt(

   3  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/poetry_plugin_export/exporter.py:95 in _export_generic_txt
        93│         )
        94│
     →  95│         for dependency_package in get_project_dependency_packages(
        96│             self._poetry.locker,
        97│             project_requires=root.all_requires,

   2  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/poetry_plugin_export/walker.py:93 in get_project_dependency_packages
        91│         selected.append(dependency)
        92│
     →  93│     for package, dependency in get_project_dependencies(
        94│         project_requires=selected,
        95│         locked_packages=repository.packages,

   1  ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/poetry_plugin_export/walker.py:121 in get_project_dependencies
       119│         )
       120│
     → 121│     nested_dependencies = walk_dependencies(
       122│         dependencies=project_requires,
       123│         packages_by_name=packages_by_name,

  RuntimeError

  Dependency walk failed at torchvision

  at ~/code/.dockerdev-data/cli-pHEz0X_z-py3.10/lib/python3.10/site-packages/poetry_plugin_export/walker.py:151 in walk_dependencies
      147│             requirement, packages_by_name, nested_dependencies
      148│         )
      149│
      150│         if not locked_package:
    → 151│             raise RuntimeError(f"Dependency walk failed at {requirement}")
      152│
      153│         if requirement.extras:
      154│             locked_package = locked_package.with_features(requirement.extras)
@dimbleby
Copy link
Contributor

dimbleby commented Aug 2, 2023

Did you really search for duplicates?!

This is duplicate of at least #163, #168, #173, #176, #183, #186, #187

@2e0byo
Copy link
Author

2e0byo commented Aug 2, 2023

Did you really search for duplicates?!

Yes. Seems I wasn't as intelligent as I could have been though.

This is duplicate of at least #163, #168, #173, #176, #183, #186, #187

#186 actually has the same setup, albeit with markers= rather than the shorthand. Ergo this really is a dupe. I don't have multiple versions of the same package in the pyproject.toml. The code path is different: 181 filters out the right version, whereas this is the inverse bug: we can't select between as decided isn't being populated correctly.

I don't want to clutter up the issue tracker with noise: feel free to close this as a dupe.

@radoering radoering added the duplicate This issue or pull request already exists label Aug 2, 2023
@radoering radoering closed this as not planned Won't fix, can't repro, duplicate, stale Aug 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists
Projects
None yet
Development

No branches or pull requests

3 participants