enviroment markers ignored when multiple version options coexist with legacy source

[seansfkelley]

• Poetry version: 1.2.1
• Python version: 3.10.6
• OS version and name: macOS 10.15.7
• pyproject.toml: https://gist.github.com/seansfkelley/fe46b800efa747f9917f9979a4757874

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Possibly related: #3561, #6215, #5506,

Issue

When I attempt poetry install on macOS, Poetry tries to install torch=1.11.0+cpu, which does not have appropriate native builds:

poetry install   
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing torch (1.11.0+cpu): Failed

  RuntimeError

  Unable to find installation candidates for torch (1.11.0+cpu)

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chooser.py:103 in choose_for
       99│ 
      100│             links.append(link)
      101│ 
      102│         if not links:
    → 103│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      104│ 
      105│         # Get the best link
      106│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      107│ 

If I remove the non-Darwin option and leave only this one:

{ version = "1.11.0", markers = "sys_platform == 'darwin'", source = "pypi" }

Poetry installs it without issue.

If I instead replace the non-PyPI option with some other, distinct option and remove the pytorch legacy source:

{ version = "1.12.0", markers = "sys_platform != 'darwin'", source = "pypi" }

(note the version and source are different from the original non-Darwin option)

I can then freely change around the markers on both versions and reinstall and it will always select the intended version according to the rules.

The behavior of apparently ignoring markers only seems to happen when the source is not PyPI.

The non-Darwin source is used in a Docker build, and, if I remove the multiple options and simply set torch = "1.11.0+cpu", I am able to correctly install torch==1.11.0+cpu in my Docker build using this source configuration.

[radoering]

Duplicates #5205 (comment). Will be fixed in upcoming 1.3 via #6679

[seansfkelley]

#6679 does not appear to resolve this. This morning I installed Poetry from master:

curl -sSL https://install.python-poetry.org | python3 - --git https://github.com/python-poetry/poetry.git@master

The self-reported version is only 1.3.0.dev0 and doesn't specify the commit hash. I assume this includes the fix, but I don't know how to check which exact commit is running.

Using the same pyproject.toml linked in the original comment:

$ poetry lock -vvv 
Loading configuration file /Users/seankelley/Library/Preferences/pypoetry/config.toml
Adding repository pytorch (https://download.pytorch.org/whl) and setting it as secondary
Creating virtualenv root-2Uq3A66e-py3.10 in /Users/seankelley/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/seankelley/Library/Caches/pypoetry/virtualenvs/root-2Uq3A66e-py3.10
Updating dependencies
Resolving dependencies...
   1: fact: root is 0.1.0
   1: derived: root    
   0: Duplicate dependencies for torch                                                                                                                        
   0: Different requirements found for torch (1.11.0+cpu) with markers sys_platform != "darwin" and torch (1.11.0) with markers sys_platform == "darwin".
   1: Version solving took 0.003 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('root', '0.1.0'): {'torch': <Dependency torch (==1.11.0+cpu)>}}).
   1: fact: root is 0.1.0
   1: derived: root
   1: fact: root depends on torch (1.11.0+cpu)
   1: selecting root (0.1.0)                                                                                                                                  
   1: derived: torch (==1.11.0+cpu)                                                                                                                           
   1: fact: torch (1.11.0+cpu) depends on typing-extensions (*)
   1: selecting torch (1.11.0+cpu)
   1: derived: typing-extensions
<c1>Source (pytorch):</c1> Authorization error accessing https://download.pytorch.org/whl/typing-extensions/
   1: selecting typing-extensions (4.3.0)
   1: Version solving took 0.538 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('root', '0.1.0'): {'torch': <Dependency torch (==1.11.0)>}}).
   1: fact: root is 0.1.0
   1: derived: root
   1: fact: root depends on torch (1.11.0)
   1: selecting root (0.1.0)
   1: derived: torch (==1.11.0)
   1: fact: torch (1.11.0) depends on typing-extensions (*)
   1: selecting torch (1.11.0)                                                                                                                                
   1: derived: typing-extensions
<c1>Source (pytorch):</c1> Authorization error accessing https://download.pytorch.org/whl/typing-extensions/
   1: selecting typing-extensions (4.3.0)
   1: Version solving took 0.136 seconds.
   1: Tried 1 solutions.
   0: Complete version solving took 0.677 seconds with 2 overrides
   0: Resolved with overrides: ({Package('root', '0.1.0'): {'torch': <Dependency torch (==1.11.0+cpu)>}}), ({Package('root', '0.1.0'): {'torch': <Dependency torch (==1.11.0)>}})

Writing lock file

$ poetry install
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing typing-extensions (4.3.0)
  • Installing torch (1.11.0+cpu): Failed

  RuntimeError

  Unable to find installation candidates for torch (1.11.0+cpu)

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chooser.py:103 in choose_for
       99│ 
      100│             links.append(link)
      101│ 
      102│         if not links:
    → 103│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      104│ 
      105│         # Get the best link
      106│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      107│ 

The lockfile format has changed a lot and it does seem to keep them separate as indicated in #6679, but the net effect is the same.

poetry.lock

# This file is automatically @generated by Poetry and should not be changed by hand.

[[package]]
name = "torch"
version = "1.11.0"
description = "Tensors and Dynamic neural networks in Python with strong GPU acceleration"
category = "main"
optional = false
python-versions = ">=3.7.0"

[[package.files]]
file = "torch-1.11.0-cp310-cp310-manylinux1_x86_64.whl"
hash = "sha256:62052b50fffc29ca7afc0c04ef8206b6f1ca9d10629cb543077e12967e8d0398"

[[package.files]]
file = "torch-1.11.0-cp310-cp310-manylinux2014_aarch64.whl"
hash = "sha256:866bfba29ac98dec35d893d8e17eaec149d0ac7a53be7baae5c98069897db667"

[[package.files]]
file = "torch-1.11.0-cp310-cp310-win_amd64.whl"
hash = "sha256:951640fb8db308a59d9b510e7d1ad910aff92913323bbe4bc75435347ddd346d"

[[package.files]]
file = "torch-1.11.0-cp310-none-macosx_10_9_x86_64.whl"
hash = "sha256:5d77b5ece78fdafa5c7f42995ff9474399d22571cd6b2de21a5d666306a2ff8c"

[[package.files]]
file = "torch-1.11.0-cp310-none-macosx_11_0_arm64.whl"
hash = "sha256:b5a38682769b544c875ecc34bcb81fbad5c922139b61319aacffcfd8a32f528c"

[[package.files]]
file = "torch-1.11.0-cp37-cp37m-manylinux1_x86_64.whl"
hash = "sha256:f82d77695a60626f2b7382d85bc566de8a6b3e50d32080755abc040db802e419"

[[package.files]]
file = "torch-1.11.0-cp37-cp37m-manylinux2014_aarch64.whl"
hash = "sha256:b96654d42566080a134e784705f33f8536b3b95b5dcde357ed7879b1692a5f78"

[[package.files]]
file = "torch-1.11.0-cp37-cp37m-win_amd64.whl"
hash = "sha256:8ee7c2e8d7f7020d5bfbc1bb91b9591044c26bbd0cee5e4f694cfd7ed8649260"

[[package.files]]
file = "torch-1.11.0-cp37-none-macosx_10_9_x86_64.whl"
hash = "sha256:6860b1d1bf0bb0b67a6bd47f85a0e4c825b518eea13b5d6101999dbbcbd5bc0c"

[[package.files]]
file = "torch-1.11.0-cp38-cp38-manylinux1_x86_64.whl"
hash = "sha256:4322aa29f50da7f404db06cdf30896ea67b09f673af4a985afc7162bc897864d"

[[package.files]]
file = "torch-1.11.0-cp38-cp38-manylinux2014_aarch64.whl"
hash = "sha256:e4d2e0ddd652f30e94cff750220324ec45705d4ecc69658f773b3cb1c7a28dd0"

[[package.files]]
file = "torch-1.11.0-cp38-cp38-win_amd64.whl"
hash = "sha256:34ce5ea4d8d85da32cdbadb50d4585106901e9f8a3527991daa70c13a09de1f7"

[[package.files]]
file = "torch-1.11.0-cp38-none-macosx_10_9_x86_64.whl"
hash = "sha256:0ccc85cd06227a3edf809e2c795fd5762c3d4e8a38b5c9f744c6e7cf841361bb"

[[package.files]]
file = "torch-1.11.0-cp38-none-macosx_11_0_arm64.whl"
hash = "sha256:c1554e49d74f1b2c3e7202d77056ba2dd7465437585bac64062b580f714a44e9"

[[package.files]]
file = "torch-1.11.0-cp39-cp39-manylinux1_x86_64.whl"
hash = "sha256:58c7814502b1c129a650d7092033bbb0bbd64faf1a7941631aaa1aeaddc37570"

[[package.files]]
file = "torch-1.11.0-cp39-cp39-manylinux2014_aarch64.whl"
hash = "sha256:831cf588f01dda9409e75576741d2823453990dee2983d670f2584b37a01adf7"

[[package.files]]
file = "torch-1.11.0-cp39-cp39-win_amd64.whl"
hash = "sha256:44a1d02fd20f827f0f36dc26fdcfc45e793806a6ad52769a22260655a77a4369"

[[package.files]]
file = "torch-1.11.0-cp39-none-macosx_10_9_x86_64.whl"
hash = "sha256:50fd9bf85c578c871c28f1cb0ace9dfc6024401c7f399b174fb0f370899f4454"

[[package.files]]
file = "torch-1.11.0-cp39-none-macosx_11_0_arm64.whl"
hash = "sha256:0e48af66ad755f0f9c5f2664028a414f57c49d6adc37e77e06fe0004da4edb61"

[package.dependencies]
typing-extensions = "*"

[[package]]
name = "torch"
version = "1.11.0+cpu"
description = "Tensors and Dynamic neural networks in Python with strong GPU acceleration"
category = "main"
optional = false
python-versions = ">=3.7.0"

[[package.files]]
file = "torch-1.11.0+cpu-cp310-cp310-linux_x86_64.whl"
hash = "sha256:32fa00d974707c0183bc4dd0c1d69e853d0f15cc60f157b71ac5718847808943"

[[package.files]]
file = "torch-1.11.0+cpu-cp310-cp310-win_amd64.whl"
hash = "sha256:bd984fa8676b2f7c9611b40af3a7c168fb90be3e29028219f822696bb357f472"

[[package.files]]
file = "torch-1.11.0+cpu-cp37-cp37m-linux_x86_64.whl"
hash = "sha256:50008b82004b9d91e036cc199a57f863b6f8978b8a222176f9a4435fce181dd8"

[[package.files]]
file = "torch-1.11.0+cpu-cp37-cp37m-win_amd64.whl"
hash = "sha256:7bbd8b77a59e628a7cb84289a3a26adc7e28dd7213c7f666537f26e714fb1721"

[[package.files]]
file = "torch-1.11.0+cpu-cp38-cp38-linux_x86_64.whl"
hash = "sha256:22997df8f3a3f9faed40ef9e7964d1869cafa0317cc4a5b115bfdf69323e8884"

[[package.files]]
file = "torch-1.11.0+cpu-cp38-cp38-win_amd64.whl"
hash = "sha256:0dbdddc7452a2c42250df369e4968b62589ab0ac1b9d14e27701eb4fc3839ad1"

[[package.files]]
file = "torch-1.11.0+cpu-cp39-cp39-linux_x86_64.whl"
hash = "sha256:544c13ef120531ec2f28a3c858c06e600d514a6dfe09b4dd6fd0262088dd2fa3"

[[package.files]]
file = "torch-1.11.0+cpu-cp39-cp39-win_amd64.whl"
hash = "sha256:7198bf5c69464459bd79526c6a4eaad2806db886443ee2f4e8e7a492bccf03ef"

[package.dependencies]
typing-extensions = "*"

[package.source]
type = "legacy"
url = "https://download.pytorch.org/whl"
reference = "pytorch"

[[package]]
name = "typing-extensions"
version = "4.3.0"
description = "Backported and Experimental Type Hints for Python 3.7+"
category = "main"
optional = false
python-versions = ">=3.7"

[[package.files]]
file = "typing_extensions-4.3.0-py3-none-any.whl"
hash = "sha256:25642c956049920a5aa49edcdd6ab1e06d7e5d467fc00e0506c44ac86fbfca02"

[[package.files]]
file = "typing_extensions-4.3.0.tar.gz"
hash = "sha256:e6d2677a32f47fc7eb2795db1dd15c1f34eff616bcaf2cfb5e997f854fa1c4a6"

[metadata]
lock-version = "2.0"
python-versions = "^3.10"
content-hash = "0adf8e52b6bf98c0eb83b81ef7d74dda816801f3b07f34d126de0c7fb03ba08d"

[dimbleby]

1.11.0 and 1.11.0+cpu are too similar: local versions are supposed to be fully compatible with the original.

So at install time when looking in the lockfile for something that satisfies 1.11.0, poetry is happy to choose 1.11.0+cpu: that's supposed to be compatible with 1.11.0, and also more recent.

I don't see anything much to be done about this.

[radoering]

But shouldn't we consider the locked package source?

[dimbleby]

At installation time poetry is presented with the following puzzle: "install torch=1.11.0, the options available to you are per this lockfile".

Considering only version constraints and markers the 1.11.0+cpu package is then the best solution.

[seansfkelley]

I agree that PyTorch's packaging is a bit odd and the whole cpu alternate build would maybe benefit from being specified using extras instead. That said, the way I understand markers (at least in this case) is that this version specification should be treated as if it doesn't exist for the purposes of installing on macOS.

Edit to respond:

Considering only version constraints and markers the 1.11.0+cpu package is then the best solution.

It doesn't seem to be considering the markers, because +cpu is not a legal choice on my system, right?

[neersighted]

The intention was to keep them separate as per #6679 -- they are interchangeable versions, but I should be able to specify two identical package specs from different sources, and get the correct version from the specified source based on my multiple constraint markers.

[dimbleby]

hmm, maybe some sort of fix around https://github.com/python-poetry/poetry/blob/master/src/poetry/puzzle/provider.py#L860-L864 could work

[dimbleby]

maybe as simple as

--- a/src/poetry/puzzle/provider.py
+++ b/src/poetry/puzzle/provider.py
@@ -859,7 +859,7 @@ class Provider:
                 # the complete_name (including features).
                 dependency.name == package.name
                 and (
-                    dependency.source_type is None
+                    (dependency.source_type is None and dependency.source_name is None)
                     or dependency.is_same_source_as(package)
                 )
                 and dependency.constraint.allows(package.version)

??

where the condition being tweaked represents "the dependency doesn't care from what source it is satisfied"

But I'll leave it to someone more motivated to add a unit test proving the point

[dimbleby]

actually I think it's all a bigger mess than that

• what a Dependency calls source_name is stored as source_reference on a Package. Probably the source_name attribute should be removed altogether, in favour of source_reference, so that these became consistent.
• per the not-great fix above, a Dependency with an explicit source_name is not universally satisfied by any package
• is_same_source_as() ought to know something about all of this - it currently believes that if both packages have source type None then "no need to check further"
• using source type None to mean "I don't care" on dependencies but "pypi" on packages is pretty confusing

ugh...

[radoering]

The other PR is still required for a fix.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.