Alternative Source Definitions Within Path Dependencies Are Not Used

[gradylynn]

• Poetry version: 1.2.1
• Python version: 3.8.8
• OS version and name: macOS 11.7
• pyproject.toml: Here is a small repo I made to demonstrate the bug.

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Here is my debug mode output

glynn@blah path-alt-sources-poetry-bug % poetry -vvv install
Loading configuration file /Users/glynn/Library/Preferences/pypoetry/config.toml
Creating virtualenv the-best-codebase-ever-HmrV7-av-py3.8 in /Users/glynn/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/glynn/Library/Caches/pypoetry/virtualenvs/the-best-codebase-ever-HmrV7-av-py3.8
Updating dependencies
Resolving dependencies...
   1: fact: the-best-codebase-ever is 0.1.0
   1: derived: the-best-codebase-ever
   1: fact: the-best-codebase-ever depends on my_package (0.0.1)
   1: selecting the-best-codebase-ever (0.1.0)
   1: derived: my_package (0.0.1) @ my_package
   1: fact: my-package (0.0.1) depends on roslz4 (^1.14.0)
   1: selecting my-package (0.0.1 /Users/glynn/Desktop/path-alt-sources-poetry-bug/my_package)
   1: derived: roslz4 (>=1.14.0,<2.0.0)
   1: fact: roslz4 doesn't exist
   1: conflict: roslz4 doesn't exist
   1: derived: not roslz4 (>=1.14.0,<2.0.0)
   1: conflict: my-package (0.0.1) depends on roslz4 (^1.14.0)
   1: ! not roslz4 (>=1.14.0,<2.0.0) is satisfied by not roslz4 (>=1.14.0,<2.0.0)
   1: ! which is caused by "roslz4 doesn't exist"
   1: ! thus: my-package is forbidden
   1: ! my-package (0.0.1) @ file:///Users/glynn/Desktop/path-alt-sources-poetry-bug/my_package is satisfied by my_package (0.0.1) @ my_package
   1: ! which is caused by "the-best-codebase-ever depends on my_package (0.0.1) @ my_package"
   1: ! thus: version solving failed
   1: Version solving took 0.060 seconds.
   1: Tried 1 solutions.

  Stack trace:

  4  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/puzzle/solver.py:151 in _solve
      149│ 
      150│         try:
    → 151│             result = resolve_version(
      152│                 self._package, self._provider, locked=locked, use_latest=use_latest
      153│             )

  3  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/mixology/__init__.py:24 in resolve_version
       22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       23│ 
    →  24│     return solver.solve()
       25│ 

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/mixology/version_solver.py:126 in solve
      124│             next: str | None = self._root.name
      125│             while next is not None:
    → 126│                 self._propagate(next)
      127│                 next = self._choose_package_version()
      128│ 

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/mixology/version_solver.py:165 in _propagate
      163│                     # where that incompatibility will allow us to derive new assignments
      164│                     # that avoid the conflict.
    → 165│                     root_cause = self._resolve_conflict(incompatibility)
      166│ 
      167│                     # Back jumping erases all the assignments we did at the previous

  SolveFailure

  Because my-package (0.0.1) @ file:///Users/glynn/Desktop/path-alt-sources-poetry-bug/my_package depends on roslz4 (^1.14.0) which doesn't exist, my-package is forbidden.
  So, because the-best-codebase-ever depends on my_package (0.0.1) @ my_package, version solving failed.

  at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/mixology/version_solver.py:364 in _resolve_conflict
      360│             )
      361│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      362│             self._log(f"! thus: {incompatibility}")
      363│ 
    → 364│         raise SolveFailure(incompatibility)
      365│ 
      366│     def _choose_package_version(self) -> str | None:
      367│         """
      368│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  Stack trace:

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/commands/command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/console/commands/install.py:145 in handle
       143│         self.installer.verbose(self.io.is_verbose())
       144│ 
     → 145│         return_code = self.installer.run()
       146│ 
       147│         if return_code != 0:

   3  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/installer.py:114 in run
       112│             self._execute_operations = False
       113│ 
     → 114│         return self._do_install()
       115│ 
       116│     def dry_run(self, dry_run: bool = True) -> Installer:

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248│         else:
       249│             self._io.write_line("Installing dependencies from lock file")

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/puzzle/solver.py:73 in solve
        71│         with self._provider.progress():
        72│             start = time.time()
     →  73│             packages, depths = self._solve(use_latest=use_latest)
        74│             end = time.time()
        75│ 

  SolverProblemError

  Because my-package (0.0.1) @ file:///Users/glynn/Desktop/path-alt-sources-poetry-bug/my_package depends on roslz4 (^1.14.0) which doesn't exist, my-package is forbidden.
  So, because the-best-codebase-ever depends on my_package (0.0.1) @ my_package, version solving failed.

  at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/puzzle/solver.py:159 in _solve
      155│             packages = result.packages
      156│         except OverrideNeeded as e:
      157│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      158│         except SolveFailure as e:
    → 159│             raise SolverProblemError(e)
      160│ 
      161│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      162│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      163│ 

Issue

This relates to installing path-dependencies which use non-pypi sources for its dependencies. I try to describe the issue in this repo (the same as what's mentioned above for the pyproject.toml prompt).

Setup

I have a project that contains a package I developed within it, structured as below:

.
├── README.md
├── my_package
│   ├── my_package
│   │   ├── __init__.py
│   │   └── module.py
│   └── pyproject.toml
├── pyproject.toml
└── script.py

The pyproject.toml file within ./my_package specifies a non-pypi dependency using [[tool.poetry.source]].
I specify my_package as a path dependency in the project's (top-level) pyproject.toml. When I run poetry install from the project's directory, I expect my_package to install it's non-pypi dependencies with the sources defined in it's pyproject.toml. But this fails with the following error:

Because my-package (0.0.1) @ file://p/a/t/h/poetry-bug-path-alt-sources/my_package depends on roslz4 (^1.14.0) which doesn't exist, my-package is forbidden.
So, because the-best-codebase-ever depends on my_package (0.0.1) @ my_package, version solving failed.

Workaround (ish)

However, when I define the same non-pypi [[tool.poetry.source]] in the project's (top-level) pyproject.toml, everything installs as expected. This leads me to believe that poetry does not use sources defined in a path-dependency's pyproject.toml, but only uses the sources defined in the top-level pyproject.toml when installing the dependencies of a path-dependency.

This seems like a bug because a project should be agnostic towards the non-pypi sources used by its dependencies.

[neersighted]

This is currently by design. You are essentially encountering the boundary of the "Poetry-only" world and the wider Python packaging ecosystem.

Poetry is actually an interoperable tool that is compatible (or strives to be) with the rest of the Python packaging ecosystem. We have cross-tool standards like PEP 517 so that you can consume Poetry packages (or have Poetry consume other packages) regardless of the tooling on the other end.

Now, it's true that Poetry does not use the PEP 517 API to build other projects that use Poetry when they are path dependencies -- however, we have, up until this point, attempted to avoid introducing magical behaviors around this that violate what is possible with a purely PEP 517-driven interaction.

Namely, a PEP 517 build system outputs a list of requirements to the caller, but does not install the requirements, or communicate information on how they are installed in the dependency itself.

Now, if you specify [[tool.poetry.source]] in the parent, you will make that repository available for consideration when resolving the tree in the parent project. Note that there are some sharp edges/tradeoffs here as Poetry originally took the approach of pip and treated all sources more-or-less equally. We're working on a new set of semantics at #6713 that should help with this.

It seems like what you are looking for is better support for monorepos/workspaces/subprojects, where path dependencies are explicitly allowed to be 'Poetry-aware' and we can 'leak' information across the larger project hierarchy, in ways that would otherwise be surprising and undesirable. That is tracked in #2270 -- and there is a user working on introducing this into Poetry sooner than later, though we have yet to do any rigorous review of the design or code.

In short, this is intended and part of how the Python packaging world works -- hopefully that explanation makes sense, and you know where future work is going/how you can help shape it, if you are interested.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.