Ability to specify a "target" for non-package mode

[david-waterworth]

Issue Kind

Brand new capability

Description

I regularly create "non-package" ML projects, intended to train models, run experiments or perform analysis. Often this requires using packages such as sentence-transformers that depends on torch which needs to be installed from different sources depending on whether an accelerator is present (i.e. a GPU) or not. So there's (at least) two different torch packages that satisfy the sentence-transformers requirement but depending on the target machine only 1 will can be used.

I've been following discussions such as #6409 (comment) but as far as I can see there isn't a way of specifying that torch is required, but based on some "extra" or "environment marker" install a specific version.

The closest I've got is:

[tool.poetry]
package-mode = false

[tool.poetry.dependencies]
python = "^3.10"
torch = [
    { platform = "darwin", version = "2.3.0", source = "pytorch-cpu" },
    { platform = "linux", version = "2.3.0+cu121", source = "pytorch+cu121" },
]
sentence-transformers = "^3.0.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[[tool.poetry.source]]
name = "pytorch+cu121"
url = "https://download.pytorch.org/whl/cu121"
priority = "explicit"

[[tool.poetry.source]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
priority = "explicit"

This will install the cuda version on linux and cpu version on macos which covers some of my use case - but there's a bunch of others, such as running in a dev container that has linux but not cuda, or a windows machine with AMD (ROCm).

I've tried using extra's but they don't work as expected - i.e. this example #6409 (comment) results in poetry installing the GPU version of CUDA, then uninstalling it, then installing it each time you run poetry install

I'm not proposing any changes in package mode, but in non-package mode it would be nice if we could specify some sort of custom "target" environment marker that allows installation of 1 of multiple sources. I'd expect it would be explicitly supplied as an argument to poetry install

Another example would be the faiss-cpu and faiss-gpu packages - I want to specify that I need one of them, and which based on some argument.

Impact

This allows a team of data scientists to use Poetry to manage dependencies for non-package mode repositories using different os (darwin, linux, win32) and accelerators (AMD, NVidia, CPU) with poetry

Workarounds

The example above partially works, it installs the GPU version of torch on Linux and the CPU verison on MacOS but doesn't allow for CPU on linux etc.

[Secrus]

While we understand the issues with accelerators in ML packages, that is not an issue we will solve in Poetry, it should be fixed globally on a packaging level, with proper support for packaging markers for GPU etc.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.