Handling Platform-Specific Dependency Conflicts in Poetry

[hugocool]

Issue Kind

Change in current behaviour

Description:

I have encountered an issue with Poetry's handling of platform-specific dependencies, particularly when using environment markers to resolve conflicts between incompatible packages. The specific scenario involves using vllm on Linux and llama-cpp-python on macOS. The issue arises because torch version 2.3.0 is required on Linux, but it is not installable on macOS, where torch version 2.2.0 is needed instead. Additionally, vllm is not installable on macOS and its compatible version with torch 2.2.0 is incompatible with pydantic 2.0.0.

Background:

• Primary Dependencies:
  • pydantic version ^2.0.0 (required on both macOS and Linux).
  • torch version 2.3.0 for Linux and version 2.2.0 for macOS (due to lack of wheels for 2.3.0 on macOS and issues with lower versions).
  • vllm version 0.5.0.post1 for Linux (optional and incompatible with macOS).

Problem:

Despite using environment markers to specify platform-specific dependencies, Poetry fails to resolve dependencies when there are conflicts across platforms. For example, specifying vllm with markers = "sys_platform == 'linux'" should ideally exclude it from macOS dependency resolution. However, Poetry still attempts to resolve vllm for macOS, leading to version conflicts with torch and making the entire dependency resolution process fail.

Detailed Steps and Attempts:

1. Initial Configuration:

   [tool.poetry.dependencies]
   python = ">=3.9.10,<3.11"
   pydantic = "^2.0.0"
   llama-cpp-python = { version = "0.2.79", optional = true }
   torch = [
       {version = "2.3.0", markers = "sys_platform == 'linux'"},
       {version = "2.2.0", markers = "sys_platform == 'darwin'"}
   ]
   vllm = {version = "0.5.0.post1", markers = "sys_platform == 'linux'", optional = true}
   
   [tool.poetry.extras]
   mac = ["llama-cpp-python"]
   linux = ["vllm"]

2. Attempted Solution with Groups:

   [tool.poetry.group.mac]
   optional = true
   
   [tool.poetry.group.linux]
   optional = true
   
   [tool.poetry.group.mac.dependencies]
   llama-cpp-python = "0.2.79"
   
   [tool.poetry.group.linux.dependencies]
   vllm = { version = "0.5.0.post1", markers = "sys_platform == 'linux'", optional = true }

   Result:

   poetry lock --no-update
   Resolving dependencies... (1.1s)
   
   Because vllm (0.5.0.post1) depends on torch (2.3.0)
   and project depends on torch (2.2.0), vllm is forbidden.
   So, because project depends on vllm (0.5.0.post1), version solving failed.
   

3. Another Attempt with Markers:

   vllm = [
       {version = "0.5.0.post1", markers = "sys_platform != 'darwin'", optional = true}
   ]

   Result:

   poetry lock --no-update
   Resolving dependencies... (1.2s)
   
   Because vllm (0.5.0.post1) depends on torch (2.3.0)
   and project depends on torch (2.2.0), vllm is forbidden.
   So, because project depends on vllm (0.5.0.post1), version solving failed.
   

Expected Behavior:

When using environment markers such as sys_platform == 'linux', Poetry should completely exclude vllm from the resolution process on macOS. This means:

• vllm should not influence or cause conflicts in the dependency resolution on macOS.
• Dependencies specified with platform markers should respect the platform constraints during resolution, particularly for optional dependencies.

Proposed Solution:

Introduce a mechanism in Poetry to truly exclude dependencies based on platform markers during the resolution process. This could be an enhancement to the existing marker handling to ensure that platform-specific optional dependencies do not interfere with the resolution on incompatible platforms.

Impact

This issue impacts users who manage multi-platform projects with platform-specific dependencies. It is particularly crucial for projects that rely on different packages on macOS and Linux due to compatibility issues. The inability to exclude certain dependencies based on the platform during resolution can lead to unresolved conflicts, hindering development and deployment processes.

Addressing this issue would allow for more precise and flexible dependency management, ensuring that platform-specific constraints are respected. This improvement would enhance the user experience by preventing unnecessary conflicts, enabling smoother workflows, and ensuring that developers can use the appropriate packages for each platform without manual intervention or complex workarounds.

Workarounds

Workarounds

1. Using Alternative Dependency Configurations:

One workaround is to adjust the dependency configurations as follows:

[tool.poetry.dependencies]
python = ">=3.9.10,<3.11"
pydantic = "^2.0"
llama-cpp-python = { version = "0.2.79", optional = true }
torch = [
    {version = "2.3.0", markers = "sys_platform == 'linux'"},
    {version = "^2.1, !=2.3.0", markers = "sys_platform == 'darwin'"}
]

[tool.poetry.group.mac]
optional = true

[tool.poetry.group.linux]
optional = true

[tool.poetry.group.mac.dependencies]
llama-cpp-python = "0.2.79"

[tool.poetry.group.linux.dependencies]
vllm = [
    {version = "0.5.0.post1", markers = "sys_platform != 'darwin'", optional = true},
    {version = "*", markers = "sys_platform == 'darwin'", optional = true}
]

This configuration resolves to a very early version of vllm that barely specified any constraints. While this may work, it is not ideal as it relies on a very old version of vllm.

2. Using a Poetry Plugin:

A better workaround involves creating a Poetry plugin that conditionally excludes a specific package from the dependency resolution process based on the platform. Below is an example of such a plugin:

from poetry.plugins.plugin import Plugin
from poetry.poetry import Poetry
from poetry.repositories.pool import Pool
from poetry.repositories.repository import Repository
from poetry.utils.env import Env

class ExcludeDependencyPlugin(Plugin):
    def activate(self, poetry: Poetry, env: Env):
        if env.platform == "darwin":
            # Exclude vllm from the dependencies
            original_pool = poetry.pool
            new_pool = Pool()
            for repository in original_pool.repositories:
                if isinstance(repository, Repository):
                    new_repo = Repository(repository.name)
                    for package in repository.packages:
                        if package.name != "vllm":
                            new_repo.add_package(package)
                    new_pool.add_repository(new_repo)
                else:
                    new_pool.add_repository(repository)
            poetry.pool = new_pool

This plugin modifies the dependency pool during resolution to exclude vllm on macOS, preventing any related conflicts. This approach ensures that platform-specific dependencies are respected, improving the overall dependency management experience in multi-platform projects.

[dimbleby]

looks duplicate in the #5506 family

[hugocool]

i wouldnt say its a duplicate, in my case i would argue that one can expect it should be possible for a package to only be installed for a specific sys_platform. And the docs dont say otherwise, nor does logic dictate that when we specify that when the sys_platform != darwin, that package is still causing resolving issues.

[dimbleby]

i wouldnt say its a duplicate

it still is, though!