Picking wrong platform installing a dependency with multiple constraints

[leon19]

• I am on the 1.0.5 Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04
• Poetry version: 1.0.5
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/leon19/42bf22ddc791e0500c614d461c32a486

Issue

Markers are not working properly with multiple constraints dependencies as stated in the docs

I am trying to install a dependency from an URL based on the platform, different URLs for every platform (markers = "sys_platform == 'linux'") and poetry is picking the URL from the platform darwin when I'm under a linux platform

I have not tested yet what happens on a Darwin platform. Will update the issue when I do

Update:

If I reverse the order of the dependencies, the first line the darwin url, it works on Linux. The same behaviour applies on Darwin

[abn]

@leon19 try chaging it to the following (this is only a partial workaround - see below).

guildai = [
  {url = "https://files.pythonhosted.org/packages/10/ed/ffc1a004da1a272131e776f4b1d24d980b1dbb25b3bbaea4aff5ff7ef538/guildai-0.7.0rc7-cp37-cp37m-manylinux1_x86_64.whl", platform = "linux"},
  {url = "https://files.pythonhosted.org/packages/17/bc/2bc49e33d9343e689723c0b67aa45653d79fc4b5342c12bd7836aca58e7b/guildai-0.7.0rc7-cp37-cp37m-macosx_10_14_x86_64.whl", platform = "darwin"}
]

This will work if you are on a linux environment. I did notice that the generated lockfile will only contain the first specified url (in this case command will succeed since I was working on a linux machine and linux platform was specified first). If I were to change the order, it will fail again. However, when building the wheel generated the correct metadata.

Requires-Dist: guildai @ https://files.pythonhosted.org/packages/10/ed/ffc1a004da1a272131e776f4b1d24d980b1dbb25b3bbaea4aff5ff7ef538/guildai-0.7.0rc7-cp37-cp37m-manylinux1_x86_64.whl; sys_platform == "linux"
Requires-Dist: guildai @ https://files.pythonhosted.org/packages/17/bc/2bc49e33d9343e689723c0b67aa45653d79fc4b5342c12bd7836aca58e7b/guildai-0.7.0rc7-cp37-cp37m-macosx_10_14_x86_64.whl; sys_platform == "darwin"

From what I can tell, when markers are explicitly used the constraints get populated differently.

Issue 1: This pertains to the use of "url". When duplicate dependencies are being merged, url and markers are ignored resulting in only the first entry to be used (in this case the linux url). Ideally, the dependency graph should be branched due to the existance of the platform marker and the url should be considered when merging duplicates.

Additionally, during investigation, I had a quick look at how this was being processed. It did raise a few questions (which might be separate issue(s) to be dealt with).

When markers are explicitly provided (eg: markers = "sys_platform == 'win32'", python = ">=3.6");

Issue 2: the python version specified is ignored if markers are explicitly specified (unsure if this is expected behaviour).

poetry/poetry/packages/package.py

Lines 349 to 359 in ae6d64d

	if not markers:
	marker = AnyMarker()
	if python_versions:
	dependency.python_versions = python_versions
	marker = marker.intersect(
	parse_marker(
	create_nested_marker(
	"python_version", dependency.python_constraint
	)
	)
	)

Issue 3: sole use of markers causes solver to be stuck in an infinite loop.
As an example, this works.

pypiwin32 = [
    { version = "220", platform = "win32", python = ">=3.6"},
    { version = "219", platform = "win32", python = "<3.6"}
]

However, this causes an infinite loop.

pypiwin32 = [
    { version = "220", markers = "sys_platform == 'win32' and python_version >= '3.6'"},
    { version = "219", markers = "sys_platform == 'win32' and python_version < '3.6'"}
]

@finswimmer would be good to get your thoughts on the above.

[leon19]

@abn I tried using platform instead of markers as suggested but it still does not work for me

Note: If I reverse the order of the dependencies, the first line the darwin url, it works on linux (I haven't tested on mac yet)

[abn]

@leon19 looks like it is non-deterministic.

[breki]

I have a similar problem. python-ldap package won't build on Windows, so I have to use an unofficial wheel file. I wanted to tell Poetry that it should use the official package for Linux and the locally stored wheel for Windows:

python-ldap = [
    { markers = "sys_platform == 'linux'", version = "*" },
    { markers = "sys_platform == 'win32'", path="lib/python_ldap-3.2.0-cp36-cp36m-win_amd64.whl" }

but, judging from the poetry.lock file, it seems markers are then merged and just determine whether the library should be installed at all:

[[package]]
category = "main"
description = "Python modules for implementing LDAP clients"
marker = "sys_platform == \"linux\" or sys_platform == \"win32\""
name = "python-ldap"

I've also asked a question on StackOverflow.

[abn]

@breki try this instead.

python-ldap = [
    { platform = "linux", version = "*" },
    { platform = "win32'", path="lib/python_ldap-3.2.0-cp36-cp36m-win_amd64.whl" }
]

[breki]

@breki try this instead.

python-ldap = [
    { platform = "linux", version = "*" },
    { platform = "win32'", path="lib/python_ldap-3.2.0-cp36-cp36m-win_amd64.whl" }
]

@abn, tried that one as well, no difference.

[eterna2]

any solution so far? I am having problem because of pyTorch. They did not release a windows package. And my dev env is a crappy windows, but my production env is linux.

It is very hard to make my docker build to work with poetry.

[iramisvalentincapco]

@abn or @eterna2 any new findings or workarounds for this issue?

I am encountering the same behavior using multiple constraints for a package using git and path dependencies based on platform.

The poetry export -f requirements.txt command seems to generate the requirements in the following way: given multiple constraints, poetry will simply use the first entry in the list as the sole dependency and attach any constraints/markers onto it via a union (markers separated by "or") This means that regardless of if the given constraints should conflict (which is the intention of the different constraints), it will simply install the first listed dependency if it meets any of the given constraints.

From my understanding, this is not the expected behavior, as multiple constraints dependencies are meant to allow for defining different locations for retrieving a given dependency.

[gilbertfrancois]

I have a similar issue. In my case, I need 3 different packages of mxnet, based on platform_machine and sys_platform.

• Linux x86_64 with Cuda (from pypi)
• Linux ARMv8 with Cuda (self compiled).
• macOS x86_64 without Cuda (from pypi)

The pyproject.toml section looks like this:

mxnet = [
    {version = "~1.6.0", markers = "sys_platform == 'darwin' and platform_machine == 'x86_64'"},
    {markers = "sys_platform == 'linux' and platform_machine == 'aarch64'", path = "../dist/cp37-cp37m-linux_aarch64/mxnet-1.6.0-cp37-cp37m-linux_aarch64.whl"}
]
mxnet-cu102 = {version = "~1.6.0",   markers = "sys_platform == 'linux' and platform_machine == 'x86_64'"}

But the poetry solver returns an error:

[RecursionError]
maximum recursion depth exceeded

[stadlerb]

I also get the RecursionError like @gilbertfrancois when I specify the CPU version of PyTorch for the win32 platform and the CUDA (10.1) version for the linux2 platform (without any machine platform constraints).

[stadlerb]

I created issue #2613 for creating different variants (CPU/CUDA) for a project, which might possibly also help with some instances of this problem.

[g4b1nagy]

This has been fixed in the 1.1.X branch of Poetry. So every version starting from 1.1.0 (Oct 1, 2020) should work just fine.
Note that:

lock files generated with this release are not compatible with previous releases of Poetry.

[finswimmer]

Thanks for your response @g4b1nagy . So I will close this. If anyone disagree please leave a comment.

[ferndot]

It looks like this may have regressed in 1.1.4

[sneko]

Hi,

Having

markers = "'linux' in sys_platform and 'armv6l' in platform_machine"

does not work with Poetry v1.1.4 even if the condition returns true when using poetry run python. Maybe linked to @joshua-s ' comment.

[sinoroc]

@sneko

markers = "'linux' in sys_platform and 'armv6l' in platform_machine"

I do not think this is a valid markers notation. What error message, if any do you see?

Should probably be the following instead:

markers = "sys_platform=='linux' and platform_machine=='armv6l'"

[sinoroc]

@joshua-s
Can you show what error messages if any you see?

[sneko]

@sinoroc I used "in" operator since sometimes I get "linux2" and at start I was looking for platform machine starting with "armv".

But I could give a try to a more complex condition with only "==" matches.

[sinoroc]

@sneko

As far as I know, only exact comparison == is allowed for such markers, but honestly I have doubts now, maybe 'linux' in sys_platform is valid after all.

For sys_platform you would get linux2 on Python 2 and linux on Python 3.

• https://docs.python.org/3/library/sys.html#sys.platform
• https://www.python.org/dev/peps/pep-0508/#environment-markers

I think I would rather use platform_system == 'Linux' anyway.

[sinoroc]

@sneko

OK, I tested a bit, and apparently the '"linux" in sys_platform' notation correct:

>>> import sys
>>> sys.platform
'linux'
>>> import packaging.requirements
>>> m = packaging.requirements.Marker('"linux" not in sys_platform')
>>> m.evaluate()
False
>>> m = packaging.requirements.Marker('"linux" in sys_platform')
>>> m.evaluate()
True

but I would not recommend it:

>>> m = packaging.requirements.Marker('"linux2" in sys_platform')
>>> m.evaluate()
False
>>> m = packaging.requirements.Marker('"linu" in sys_platform')
>>> m.evaluate()
True

[Leon0402]

I tried several combinations and none worked. Among the things I tried were the following things:

torch = [
    {url = "https://download.pytorch.org/whl/cu102/torch-1.6.0-cp38-cp38-win_amd64.whl", python = "~3.8", markers = "sys_platform == 'win32'" },
    {version = "1.6.0", markers = "sys_platform != 'win32'" }
]

torch = [
    {url = "https://download.pytorch.org/whl/cu102/torch-1.6.0-cp38-cp38-win_amd64.whl", python = "~3.8", platform = "win32" },
    {version = "1.6.0", platform = "linux || darwin" }
]

I tried both and similar variations of the two above with poetry 1.1.0, 1.1.1, 1.1.2, 1.1.3 and 1.1.4

In all cases, the first entry was always used no matter what platform I was on. So on Linux it downloaded the windows wheel. And On Windows, if you changed the order, it took the instructions for linux / mac os.

Could be clarified how exactly this should work and with what poetry versions in specific. So far it failed with all versions, so I'm slightly confused why this issue has been marked as resolved.

[ferndot]

@sinoroc I ran:

poetry export --without-hashes > requirements.txt

Here's an excerpt from my pyproject.toml:

[tool.poetry.dependencies]
python = "2.7.18 || ~3.7"
cryptography = [
    { version = "2.3.1", python = "<3" },
    { version = "~3.2", python = ">=3.7" },
]

I received a requirements list with flags like this:

cryptography==2.3.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" or python_version < "3"
cryptography==3.2.1; python_version >= "3.7"

When attempting to pip install this, I get:

ERROR: Double requirement given: cryptography==3.2.1 (from -r requirements.txt (line 27)) (already in cryptography==2.3.1 (from -r requirements.txt (line 26)), name='cryptography')

[failable]

Don't see the issued get fixed in 1.1.4, I may try a lower version.

[sneko]

@sinoroc about your answer: #2138 (comment)

Even if the tests pass... I never succeeded making this working (to take in account linux and linux2):

markers = "'linux' in sys_platform"

Instead of that, the following works:

markers = "sys_platform == 'linux' or sys_platform == 'linux2'"

I can't explain why but I'm fine with that for now :)

[sinoroc]

I can't explain why but I'm fine with that for now :)

Yeah, don't use the in. If I am not mistaken, this is simply how in with strings works in Python. If the the substring is contained in the larger string then it succeeds. I am not sure why this is allowed in markers, makes me also wonder if it is just evaled or execed or something strange like that. I don't know.

[agurtovoy]

@finswimmer I'm running poetry 1.1.6, and it appears to still have problems resolving multiple constraints dependencies.

Here's my dependencies spec:

[tool.poetry.dependencies]
python = "^3.8"
torch = [
    { version = "^1.8.1", markers = "sys_platform == 'darwin'" },
    { url = "https://download.pytorch.org/whl/cu111/torch-1.8.1%2Bcu111-cp38-cp38-linux_x86_64.whl", markers = "sys_platform == 'linux'" },
]

Given the above, when I run poetry install on a Mac, it goes for the Linux wheel instead of the vanilla torch version:

$ poetry install
Creating virtualenv poetry-test in [...]/poetry-test/.venv
Updating dependencies
Resolving dependencies... (112.3s)

  SolverProblemError

  Because poetry-test depends on torch (1.8.1+cu111) which doesn't match any versions, version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Thoughts?

[versis]

@agurtovoy
Why not:

torch = [
   {markers = "sys_platform == 'linux'", url = "https://download.pytorch.org/whl/cpu/torch-1.8.1%2Bcpu-cp38-cp38-linux_x86_64.whl"},
   {markers = "sys_platform == 'darwin'", url = "https://download.pytorch.org/whl/cpu/torch-1.8.1-cp38-none-macosx_10_9_x86_64.whl"}
]

After looking at poetry.lock (removed common stuff) I see:

[[package]]
name = "torch"
version = "1.8.1"

[package.source]
type = "url"
url = "https://download.pytorch.org/whl/cpu/torch-1.8.1-cp38-none-macosx_10_9_x86_64.whl"

[[package]]
name = "torch"
version = "1.8.1+cpu"

[package.source]
type = "url"
url = "https://download.pytorch.org/whl/cpu/torch-1.8.1%2Bcpu-cp38-cp38-linux_x86_64.whl"

I'm no poetry expert, but seems good to me.

P.S. ^ in ^1.8.1 doesn't seem to be safe. With that you would allow higher version of pytorch for mac, but not for linux (which is "hardcoded" with url).

[williamfzc]

Now poetry 1.6.1
I still failed to handle different platform versions of torch with poetry.

poetry has a good conception but finally I have to give up using it in platform sensetive projects.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.