Poetry picks wrong platform when resolving a local poetry project dependency

[sbadredd-sony]

• Poetry version: 1.7.0

• Python version: 3.12.0

• OS version and name: macOS 12.5.1

• pyproject.toml: The two pyproject.toml files to recreate the issue

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I created two local poetry projects. The first (local_dependency) defines a platform-specific dependency. The second (test_project) installs the first project using the wrong platform.

Steps to reproduce

1. Create 2 empty projects local_dependency and test_project with poetry new
2. In local_dependency/pyproject.toml I have:

[tool.poetry.dependencies]
python = ">=3.10,<3.12"
torch = [
    { version = "=2.1.0+cpu", source = "torch_cpu", platform = "linux" },
    { version = "=2.1.0", source = "pypi", platform = "darwin" },
]

[[tool.poetry.source]]
name = "torch_cpu"
url = "https://download.pytorch.org/whl/cpu"
priority = "explicit"

3. In test_project/pyproject.toml I have:

[tool.poetry.dependencies]
python = ">=3.10,<3.12"
local-dependency = {path = "../local_dependency", develop = true}

[[tool.poetry.source]]
name = "torch_cpu"
url = "https://download.pytorch.org/whl/cpu"
priority = "explicit"

Note that I need to re-add the torch_cpu source otherwise poetry lock gives me Repository "torch_cpu" does not exist.. I don't know if this is another bug or expected behavior (sources not being imported from local poetry project dependencies) Edit: this part is expected behavior; see dimbleby's answer

4. In local_project/, poetry lock and poetry install run fine. Poetry uses the pypi source for darwin as expected.
5. In test_project/, poetry lock runs fine but poetry install gives a runtime error Unable to find installation candidates for torch (2.1.0+cpu). Poetry tries to install the local dependency as if it was in linux. It seems the platform tag is not transited in the correct way

[dimbleby]

sources are a poetry special: there is no way to express them in standard project metadata and so in general project A cannot know about sources defined in project B.

what you are trying to do will not work, you'll have to be explicit about such things in your top-level project

please close

[sbadredd-sony]

So I cannot use Poetry if I have a dependency that relies on an OS-specific source. Is that correct?

[dimbleby]

your top-level project cannot rely on anything your dependencies say about sources.

Your mental model should be: imagine that the dependency has been published to pypi, and all source information is therefore unavailable - because it just isn't part of the wheel metadata

[sbadredd-sony]

Thanks for the clarification! Let's say I explicitly add the source torch_cpu in every project, like above.

For my main issue: I am on darwin, why is it installing the linux dependency?

[dimbleby]

a.b.c+local is a match for a.b.c, you still are relying on the unavailable source information to disambiguate

[sbadredd-sony]

I may be misunderstanding.
Isn't the platform argument supposed to disambiguate?

    { version = "=2.1.0+cpu", source = "torch_cpu", platform = "linux" },
    { version = "=2.1.0", source = "pypi", platform = "darwin" },

[dimbleby]

The requirement =2.1.0 is met by version 2.1.0+cpu, it was the sources that allowed differentiation

[chielk]

I ran into (almost exactly) the same issue today, so I thought it best to add my comment here.

I have torch 2.1.0 in my primary source and 2.1.0+cu118 in my supplemental source.
Adding the sources resolved the issue, (but wouldn't have worked if they were hosted on the same server).

I would expect poetry to try to install from the primary source before the supplemental one, but it's not.

Something else that's new, and I think related, is that I get a lot of messages like this about my supplemental source when running poetry lock:

Source (torch): Authorization error accessing https://download.pytorch.org/whl/cu118/<dependency>/

All these dependencies are available on the primary source, so I don't think poetry should query the supplemental one (according to the documentation https://python-poetry.org/docs/repositories/#supplemental-package-sources)

This all worked as expected a few days ago, but it broke on poetry 1.5.1 (which I hadn't updated), and wasn't fixed by updating to 1.7.1. I'm running this on MacOS 14.0 (ARM) with python 3.11.4

[dimbleby]

@chielk none of that is new and it is discussed in lots of other issues, please search for duplicates.

this issue is specifically about sources that are mentioned in a dependency of the top-level project - and it should be closed, because everything here is working as expected

[sbadredd-sony]

The requirement =2.1.0 is met by version 2.1.0+cpu, it was the sources that allowed differentiation

The platform also allows differentiation, right?

Even if you remove the sources, you have

    { version = "=2.1.0+cpu", platform = "linux" },
    { version = "=2.1.0",  platform = "darwin" },

Why would poetry pick "=2.1.0+cpu" if I am on darwin? This still seems like a bug

[dimbleby]

because that's what the python packaging standards say is the correct behaviour

https://packaging.python.org/en/latest/specifications/version-specifiers/#version-matching

If the specified version identifier is a public version identifier (no local version label), then the local version label of any candidate versions MUST be ignored when matching versions.

Therefore when matching "=2.1.0" we MUST ignore the "+cpu" on the candidate version "2.1.0+cpu", and it becomes a match

[sbadredd-sony]

Ah, now I understand, thanks!

That's a tricky one. My easiest solution is probably to work with, e.g.

    { version = "=2.0.1+cpu", source = "torch_cpu", platform = "linux" },
    { version = "=2.1.0", source = "pypi", platform = "darwin" },

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.