Parse wheel requirements Requires-Dist with platform markers | Torch 2.1.1+cu121 installation problem

[MrMarvel]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the FAQ and general documentation and believe that my question is not already covered.

Feature Request

I am on Windows 10 and I don't need linux requirements.

Poetry poetry show torch shows that it need triton.


Torch 2.1.1+cu121 and similiar versions has this format of Requires-Dist.
Can you implement support for these cases?

pyproject.toml

[tool.poetry]
name = "moneyconv"
version = "0.1.0"
description = ""
authors = ["MrMarvel <seregakkk999@yandex.ru>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.11"
jupyterlab = "^4.0.9"
torch = {version = "2.1.1", source = "torch"}


[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu121"
priority = "explicit"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Unsatisfied but working temporary solution:
Downgrading to 2.0.1 with cu118. where there is no Requires-Dist for linux platform in torch package.
pyproject.toml

[tool.poetry]
name = "moneyconv"
version = "0.1.0"
description = ""
authors = ["MrMarvel <seregakkk999@yandex.ru>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.11"
jupyterlab = "^4.0.9"
torch = {version = "2.0.1", source = "torch"}


[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[MrMarvel]

I think this features is connected with #8681

[dimbleby]

poetry builds cross-platform solutions. During locking, it doesn't matter that you're not on linux: poetry will still make sure that the lockfile it builds works on linux (and elsewhere).

there are open issues - and even a pull request (#4952 probably abandoned) - suggesting a not-cross-platform option, but today it doesn't exist.

so this is working as intended; and if you think it should change then that's a duplicate

please close

[MrMarvel]

there are open issues - and even a pull request (#4952 probably abandoned) - suggesting a not-cross-platform option, but today it doesn't exist.

A: #4952 is about optimization and has another idea and resolvement. Not duplicate of it.

poetry builds cross-platform solutions. During locking, it doesn't matter that you're not on linux: poetry will still make sure that the lockfile it builds works on linux (and elsewhere).

I don't understand clearly. Poetry does general purpose of installation and organizations environments in project. In my opinion it's like pip but better. I can install package via pip pip install torch --index-url https://download.pytorch.org/whl/cu121 but can not do this via poetry because it tries to find triton for linux (while I don't need this package).
As you said poetry verify existance of linux specific package because of cross-platform usage?
But I think we need to verify linux packages only on installation on linux machines. because it will fail on other platforms. For me it sounds logically.
So there is question: Why do we verify other-platform(Linux) specific packages on adding/installing package on specific platform(Windows)?

[MrMarvel]

So there is question: Why do we verify other-platform(Linux) specific packages on adding/installing package on specific platform(Windows)?

For now I see this as bug or error of poetry logic.
Trying installing or adding linux packages for windows.

[MrMarvel]

In short, I want poetry install to install general and platform specific packages(like platform specific packages was declared in wheel files)

[dimbleby]

Ah, sorry, I misunderstood.

But then the explanation is simple: the pytorch wheel that poetry looks at when deciding torch's dependencies contains different information.

You can check for yourself, it's https://download.pytorch.org/whl/cu121/torch-2.1.1%2Bcu121-cp310-cp310-linux_x86_64.whl

you'll need to raise this with pytorch, please close

[MrMarvel]

You can check for yourself, it's https://download.pytorch.org/whl/cu121/torch-2.1.1%2Bcu121-cp310-cp310-linux_x86_64.whl

I downloaded this wheel.

Yea, In this LINUX wheel these requirements are not optional.
As I am on Windows there is another whell for this version:
https://download.pytorch.org/whl/cu121/torch-2.1.1%2Bcu121-cp311-cp311-win_amd64.whl

There is difference in wheel files between platforms.

you'll need to raise this with pytorch, please close

I have already seen issues when you said raise problem on these projects because of wheel files. I think that is not effective problem solvement. But I am not sure that is there another way to fix it.

[MrMarvel]

Next:

But then the explanation is simple: the pytorch wheel that poetry looks at when deciding torch's dependencies contains different information.

So there is new question: Windows and linux wheel files are different. Which of them is read by poetry during locking?

[dimbleby]

I have already answered this "new" question; and anyway it should by now be obvious - the dependencies that poetry reads while locking are the ones in the linux wheel

[MrMarvel]

I have already answered this "new" question; and anyway it should by now be obvious - the dependencies that poetry reads while locking are the ones in the linux wheel

OK. If poetry is CROSS-PLATFORM product, I don't understand why it uses ONLY LINUX wheel files?

[dimbleby]

poetry makes the assumption that all wheels contain the same metadata, with platform-specific variation encoded in markers. The performance penalty of downloading all wheels and trying to deal with variation between them would make poetry unusable.

pytorch has in fact been making progress towards this goal of uniform metadata - eg see the contents of that windows wheel - but plainly they aren't yet there.

Again: you'll want to report this to them, please close this issue

[MrMarvel]

poetry makes the assumption that all wheels contain the same metadata, with platform-specific variation encoded in markers. The performance penalty of downloading all wheels and trying to deal with variation between them would make poetry unusable.

Interesting. With checking only universal format wheels you need to download less files. I agree.

Then if you can not download both files in case of optimization. As new feature request, can we download current specific platform wheel file as input for uniform metadata.
EXAMPLES on different machine on LOCK process:
On LINUX: download only https://download.pytorch.org/whl/cu121/torch-2.1.1%2Bcu121-cp310-cp310-linux_x86_64.whl
On WINDOWS: download only https://download.pytorch.org/whl/cu121/torch-2.1.1%2Bcu121-cp311-cp311-win_amd64.whl.

[dimbleby]

back where we started... if and when poetry has a feature for non-cross-platform lockfiles then what you suggest makes sense. But until then poetry is committed to the assumption that it doesn't matter which wheel it uses, it must get the same answer anyway.

[MrMarvel]

But until then poetry is committed to the assumption that it doesn't matter which wheel it uses

OK. I see the direction.

If this assumption is always right there -> in every package must be ONLY 1 file of wheel per version because there is no reason to have 2nd.
If there is 2nd wheel file(just checking, not downloading) for package it's like 100% the assumption that wheel files are equal is wrong.

SO I see only 3 decisions:

1. Ignore 2nd, force people to use format, and use only wheel(only linux). It's current for poetry. Download only 1 file.
2. Use current specific wheel (1st or 2nd wheel). It has same assumption but give more freedom and resolve some problem as mine. Download only 1 file.
3. Convert both wheel specifics. But It will need to download 2 files.

[dimbleby]

I can't follow what you're saying but I don't think it matters. poetry assumes that all distributions of a particular version of a package contain the same metadata, and packages that don't fit in with this are not compatible with poetry. That is not likely to change.

Happily nearly all packages do follow this assumption so this nearly always works out fine. You've unfortunately hit one of the exceptions.

But the torch folk have been willing recently to try and do what is needed to make their wheels all contain the same metadata. I can only say - again - you should report this to them. I expect they'll want to put it right.

[MrMarvel]

I know that your rule is breaken by them.
Can you add back support for these cases (exceptions) to avoid this problem even if they break them?

3. Convert both wheel specifics. But It will need to download 2 files.

With warning like: Downloading 2 versions of wheel files, cause package torch have not equal wheels for platforms

Or maybe it's idea for plugin to support this problem. But I think it have to be in core.

[dimbleby]

Can you add back support for these cases (exceptions) to avoid this problem even if they break them?

No.

The problem that poetry has to solve is already hard enough: it becomes quite intractable if it is additionally necessary to reverse-engineer a set of markers that correspond to what we wish package authors had written in the first place.

[MrMarvel]

Ok. I think think I got answers on my questions.
Short answer to issue: Everything work as intended. Problem with platform specific packages are not regulated by poetry and just don't work.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.