Support for build variants/profiles

[stadlerb]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

For some Python packages, there are different wheel variants with identical package names, which are distinguished only based on PEP-440 local version labels (the label in 1.2.3+label). E.g. for PyTorch or mxnet, this is used to deliver different variants that use CPU or one of several CUDA versions. This is independent of system or hardware platform and Python version - PyTorch have a wheel for almost every combination of these.
For example, for PyTorch 1.5.1 with Python 3.8 on Linux there are versions pytorch-1.5.1 (for CUDA 10.2), pytorch-1.5.1+cu101 (for CUDA 10.1), pytorch-1.5.1+cu92 (for CUDA 9.2) and pytorch-1.5.1+cpu (for CPU).

As far as I know, the only way Poetry supports for switching between these variants on a given platform is to modify the pyproject.toml file, which obviously is not so optimal. Markers are not a solution here because it is not possible switch between different variants on the same machine. Neither are optional dependencies and extras, because the package names are identical for the different variants.

In the Java world, Maven offers build profiles, which allow for specifying different sets of dependencies, build plugins etc.. Maven profiles can be activated and deactivated on a per-build basis (or using triggers similar to markers, but that is not interesting for this use case) and independently of each other.
Maven profiles are a bit complex, but I think picking and modifying some of its ideas for this specific use case could be useful for Poetry.

I'd suggest the following syntax in pyproject.toml:

[tool.poetry]
name = "myproject"
version = "1.2.3"
...

[tool.poetry.dependencies]
# Global dependencies (for all profiles)
...


[[tool.poetry.profile]]
name = "cuda-10.1-backend"
# Suggested syntax for specifying different local versions for different profiles
localversion = "cuda101"

[tool.poetry.profile.dependencies]
# Specific dependencies for CUDA 10.1 profile
torch = [
 {url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-win_amd64.whl", platform="win32"},
 {url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]


[[tool.poetry.profile]]
name = "cpu-backend"
localversion = "cpu"

[tool.poetry.profile.dependencies]
# Specific dependencies for CPU profile
torch = [
 {url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-win_amd64.whl", platform="win32"},
 {url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]

There could be one lock file per profile/combination of profiles (which I think would be easier to understand and maintain, and probably also to implement) or one global lockfile which combines several profiles/profile combinations.
The profiles could be activated using a CLI switch --profile/-p for poetry update/lock/install/add/etc.;
E.g. poetry lock -p cpu-backend would only lock the versions for the CPU variant, and poetry install -p cpu-backend would install the global packages and those defined in the cpu-backend lockfile.

Another variant would be profile groups where one of several alternatives must be active, and one of them may be active by default:

[tool.poetry]
name = "myproject"
version = "1.2.3"
...

[tool.poetry.dependencies]
# Global dependencies (for all profiles)
...


[[tool.poetry.profilegroup]]
name = "backend"

# Either cuda-10.1 or cpu must be activated during lock/install/update, cpu is active by default, but remains deactivated if cuda-10.1 is activated
[[tool.poetry.profilegroup.profile]]
name = "cuda-10.1"
localversion = "cuda101"

[tool.poetry.profilegroup.profile.dependencies]
# Specific dependencies for CUDA 10.1 profile
torch = [
 {url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-win_amd64.whl", platform="win32"},
 {url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]


[[tool.poetry.profilegroup.profile]]
name = "cpu"
default = true
localversion = "cpu"

[tool.poetry.profilegroup.profile.dependencies]
# Specific dependencies for CPU profile
torch = [
 {url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-win_amd64.whl", platform="win32"},
 {url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]

For the profile group variant the syntax could be "-p backend=cpu" etc.

An alternative solutions might be allowing to specify constraints under extras, but I don't think that's the point of extras.

[tarasivashchuk]

Anything on this? I always have to create a separate script to install PyTorch because of this.

[sinoroc]

I am not familiar with that topic, but if I understood right these discussions might be related:

• Options to build the same package in different ways (with different build dependencies)?
• Idea: selector packages

[My impression is that it is not something that poetry can/should tackle on its own. I feel like there is some background work that should be done in the larger Python packaging community beforehand: standardizations, and so on.]

[jhrmnn]

A very trivial kind of "support" would be to (optionally?) not reinstall a package by install when the installed version differs from the locked one only in the build label. Then one could at least once "overwrite" a package manually, and then Poetry would leave it be. Instead with Poetry 1.1.4, this happens:

  • Updating torch (1.7.1+cu101 -> 1.7.1)

[kikohs]

After spending a couple of hours on this issue, I found a "solution" by combining Poetry and pip just for PyTorch. You don't need to specify the wheel URLs directly and thus remain cross-platform.

I'm using Poe The Poet, a nice task runner for Poetry that allows to run any arbitrary command.

[tool.poetry.dev-dependencies]
poethepoet = "^0.10.0"

[tool.poe.tasks]
force-cuda11 = "python -m pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html"

You can run:

poetry install

and then:

poe force-cuda11  # relies on pip and use PyTorch wheels repo

Hope it helps.

[sinoroc]

[tool.poe.tasks]
force-cuda11 = "pip3 install torch==1.8.0+cu111 torchvision==0.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html"

Reminder that you should always call the executable modules python -m pip, pythonX.Y -m pip, etc. instead of the (unreliable) script wrappers pip, pipX.Y, etc.
https://snarky.ca/why-you-should-use-python-m-pip/

[kikohs]

@sinoroc I'm not sure it's really needed here as everything runs into Poetry virtual env but one can never be too careful! Thanks for sharing. I edited the comment.

[sinoroc]

I'm not sure it's really needed here as everything runs into Poetry virtual env but one can never be too careful! Thanks for sharing. I edited the comment.

Absolutely, it should not be needed. But I find it is a good habit to take. Especially since in this case it is in a file, so you type the long form just once. The shorter script wrappers are good for interactive command line sessions (less characters to type).

[jhrmnn]

Relevant discussion on Python.org:

https://discuss.python.org/t/what-to-do-about-gpus-and-the-built-distributions-that-support-them/7125

[vedosis]

At least within pyproject.toml I can get just torch to install via:

[[tool.poetry.dependencies.torch]]
python = "^3.8"
markers = "platform_system == 'Linux'"
url = "https://download.pytorch.org/whl/lts/1.8/cpu/torch-1.8.1%2Bcpu-cp38-cp38-linux_x86_64.whl"

However, the version comes across as 1.8.1+cpu. So trying to install torchvision in the same way fails because:

  SolverProblemError

  Because torchvision (0.9.1+cpu) depends on torch (1.8.1)
   and librephotos depends on torch (1.8.1+cpu), torchvision is forbidden.
  So, because librephotos depends on torchvision (0.9.1+cpu), version solving failed.

Any concerns with version to be specified and for that value to override what's discovered for dependency resolution?

Alternatively, dropping +label during version resolution would work for me.

I'm happy to do the implementation if you don't have spare cycles to do it.

[kaustubhharapanahalli]

Hello, any update on this issue? I am facing the same issue.
Looks like the issue is because of considering the +cpu or +cu111 in the version as the URL is being provided.

According to PEP 440, poetry should be ignoring these local versions: https://www.python.org/dev/peps/pep-0440/#local-version-identifiers

[danmackinlay]

@kaustubhharapanahalli Support for neural net packages is one of many unresolved issues waiting in the ticket queue for poetry at the moment. If we want any of them to progress we might need to investigate volunteering as poetry maintainers.

[kaustubhharapanahalli]

@danmackinlay I see, thanks for the update.

[danmackinlay]

@kaustubhharapanahalli regarding the pull-request backlog, see #4595

[kaustubhharapanahalli]

@danmackinlay looks like the resolving of this issue might take some time if my understanding is right 😅

Actually there is a fix that was developed and PR was raised (#4221). So not really sure how long it might take as well. But I do support the governance ticket :)

[danmackinlay]

NB the hack from March to use pip to install a desired package over the poetry version still works. It has various side effects that are undesirable (e.g. it must be run each time poetry "resolves" dependencies), so I find it easier to ditch poetry entirely in favor of a virtualenv in this case, but if there are some other features of poetry that you need this might be of interest.

[mastern2k3]

this used to work on preview release, but now it stopped working
is anybody else experiencing that?

[392781]

Why doesn't poetry support something akin to --extra-index-url in pip? I think this would solve the problem (at least in the case of PyTorch).

Right now, Poetry just defaults to installing the nvidia dependencies for PyTorch regardless if your hardware requires them.