Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--find-links equivalent #1391

Closed
2 tasks done
severinsimmler opened this issue Sep 19, 2019 · 37 comments
Closed
2 tasks done

--find-links equivalent #1391

severinsimmler opened this issue Sep 19, 2019 · 37 comments
Labels
kind/feature Feature requests/implementations

Comments

@severinsimmler
Copy link

  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have searched the documentation and believe that my question is not covered.

Using:

  • Python 3.7.3
  • Poetry 1.0.0b1

Feature Request

Because the CPU version of PyTorch is not on PyPI, they recommend installing it this way:

pip install torch==1.2.0+cpu --find-links https://download.pytorch.org/whl/torch_stable.html

However, there is no way to do this with Poetry?

Pointing directly to the wheel:

poetry add https://download.pytorch.org/whl/cpu/torch-1.2.0%2Bcpu-cp36-cp36m-manylinux1_x86_64.whl

results in

Updating dependencies
Resolving dependencies... (2.0s)


Package operations: 1 install, 0 updates, 0 removals

  - Installing torch (1.2.0+cpu https://download.pytorch.org/whl/cpu/torch-1.2.0%2Bcpu-cp36-cp36m-manylinux1_x86_64.whl)

[EnvCommandError]
Command ['/home/simmler/.cache/pypoetry/virtualenvs/epss-WeFenOJ_-py3.7/bin/python', '-m', 'pip', 'install', '--no-deps', 'torch==1.2.0+cpu'] errored with the following return code 1, and output: 
Collecting torch==1.2.0+cpu
  Could not find a version that satisfies the requirement torch==1.2.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.4.1, 0.4.1.post2, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0)
No matching distribution found for torch==1.2.0+cpu

And adding https://download.pytorch.org/whl/torch_stable.html as a repository to the pyproject.toml does not work, because it (the PyTorch wheel index page) is not PEP conform.

@kasteph kasteph added kind/feature Feature requests/implementations Hacktoberfest labels Sep 26, 2019
@cc-a
Copy link

cc-a commented Oct 20, 2019

Hi. I was thinking of having a go at this for Hacktoberfest.

The approach I was thinking of:

  • Implement a variation of the LegacyRepository class
  • Extend the expected fields for the source toml section to add a flag that specifies this type of simple html repository
  • Implement the relevant logic in poetry.py to determine the correct repository class to use

Sound about right?

@severinsimmler
Copy link
Author

Sounds like a plan 👍

@woodlandhunter
Copy link

@cc-a a --find-links that is equivalent to pip's would also accept file paths (a directory). See pip docs here. We could really use this. It's also mentioned in these issues:

#558
#1289

Your comment doesn't mention this scenario, would --find-links=DIR be covered by what you are working on?

@cc-a
Copy link

cc-a commented Oct 25, 2019

I'll have a crack at it, probably in a second PR. In principle the same logic should be required just working with file paths instead of urls.

@golyalpha
Copy link

golyalpha commented Dec 22, 2019

Right now the workaround is defining remote file dependencies (in pytorch's case, platform specific).
This works for me on the current release:

poetry add https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl --platform win32

Unfortunately, this will limit you to only one platform, as you can't have the same dependency for multiple platforms with different file links.

@germn
Copy link
Contributor

germn commented Jan 11, 2020

After installing torch with

poetry add https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl

each time I add any package poetry does following:

- Updating torch (1.3.1 -> 1.3.1 https://download.pytorch.org/whl/cu101/torch-1.3.1-cp37-cp37m-win_amd64.whl)

It doesn't break anything, but takes some time.

Anyone knows a way to avoid "updating"?

@golyalpha
Copy link

golyalpha commented Jan 11, 2020

Not really, it's a workaround for a current limitation in Poetry, but until that limitation is fixed, you could probably limit the impact by using a local cache (like devpi) @germn

@allen-chin
Copy link

@germn Something like below should work fine for your case.
pyspark = {file = "pyspark-3.0.0.dev0.tar.gz"}

@golyalpha
Copy link

golyalpha commented Feb 4, 2020

@allen-chin The sideeffect of that is continual reinstallations of the package. Just because the package is local, doesn't mean Poetry won't reinstall it.

@cpbotha
Copy link

cpbotha commented Mar 17, 2020

I created a valid simple index that can be used with index-url and as a tool.poetry.source for all of the pytorch.org official packages. You can find the new index here: https://vxlabs.com/pypi/

I massaged the official PyTorch find-urls file to look more like a PEP 503 compatible simple index, but it's still pointing to all of the official packages at pytorch.org.

It's currently working like a charm in my poetry setup.

@aviramha
Copy link
Contributor

@cpbotha Plan on adding 1.5.0 soon?
BTW Any news on this issue?

@cpbotha
Copy link

cpbotha commented May 27, 2020

Currently swamped, have not had time to get back to this and probably won't for the coming weeks.

There's also the issue on pytorch's side to make available the right sort of index for their packages, see pytorch/pytorch#25639 , but no-one has had time yet to dive in.

@timorkal
Copy link

@cpbotha great job, but I have an issue when I am using your source with poetry when trying to lock.
pyproject.toml:
torch = "1.3.1+cpu"
torchvision = "0.4.2+cpu"

[SolverProblemError]
Because torchvision (0.4.2+cpu) depends on torch (1.3.1)
and package depends on torch (1.3.1+cpu), torchvision is forbidden.
So, because package on torchvision (0.4.2+cpu), version solving failed.

@lironmo
Copy link

lironmo commented Jun 28, 2020

same issue here.
@cpbotha i tried to add your private pip repository with no success:
poetry config repositories.torch-mirror https://vxlabs.com/pypi

@daniel-j-h
Copy link

The same is true for the latest stable pytorch and torchvision packages.

The officially recommended way is

pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

but without an equivalent to the -f option in poetry we can only add the wheels manually.

But adding the wheels manually fails because the solver thinks (1.5.1) != (1.5.1+cpu url)

$ poetry add --platform linux https://download.pytorch.org/whl/cpu/torchvision-0.6.1%2Bcpu-cp38-cp38-linux_x86_64.whl https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp38-cp38-linux_x86_64.whl
Skipping virtualenv creation, as specified in config file.

Updating dependencies
Resolving dependencies... (36.4s)

[SolverProblemError]
Because myproject depends on torchvision (0.6.1+cpu url) which depends on torch (1.5.1), torch is required.
So, because myproject depends on torch (1.5.1+cpu url), version solving failed.

Should we split this into a new issue separate from the -f issue, though?

And adding here, what's the recommended way to install pytorch and torchvision with poetry now? If it's not possible (other than going through some 3rd party index as described above) that would be quite unfortunate.

@severinsimmler
Copy link
Author

Most of us only have this problem because of PyTorch, right? To be honest, since I opened this issue I no longer see the problem here with Poetry, but with PyTorch. Why the hell isn't a big project like PyTorch published on a PEP-conform index? I opened an issue (pytorch/pytorch#26340) some time ago -- so far without any concrete reaction.

I actually like the strict policy of Poetry. PEP exists for a reason.

@kousu
Copy link

kousu commented Aug 15, 2020

@severinsimmler I'm having the same issue (spinalcordtoolbox/spinalcordtoolbox#1526 (comment)). Thanks a lot for trying to get the ball rolling with pytorch!

I agree poetry should stick close to the core pypa specs, and they want to kill --find-links and dependency_links in favour of https://www.python.org/dev/peps/pep-0508/#environment-markers (pypa/pip#6162, pypa/pip#4187) so you shouldn't try to reintroduce that feature here. But PEP 508's URL mode doesn't cover wheels with compiled platform-specific versions :/ so we're stuck.

But if pytorch got themselves together and published to pypi this would be a non-issue.

@Korijn
Copy link

Korijn commented Sep 12, 2020

The same is true for the latest stable pytorch and torchvision packages.

The officially recommended way is

pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

but without an equivalent to the -f option in poetry we can only add the wheels manually.

But adding the wheels manually fails because the solver thinks (1.5.1) != (1.5.1+cpu url)

$ poetry add --platform linux https://download.pytorch.org/whl/cpu/torchvision-0.6.1%2Bcpu-cp38-cp38-linux_x86_64.whl https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp38-cp38-linux_x86_64.whl
Skipping virtualenv creation, as specified in config file.

Updating dependencies
Resolving dependencies... (36.4s)

[SolverProblemError]
Because myproject depends on torchvision (0.6.1+cpu url) which depends on torch (1.5.1), torch is required.
So, because myproject depends on torch (1.5.1+cpu url), version solving failed.

Should we split this into a new issue separate from the -f issue, though?

And adding here, what's the recommended way to install pytorch and torchvision with poetry now? If it's not possible (other than going through some 3rd party index as described above) that would be quite unfortunate.

See #2543 for the separate issue

For what it's worth, I don't think poetry needs this --find-links feature. Rather, the pytorch team needs to adopt proper packaging practices..

@golyalpha
Copy link

@Korijn The pytorch team is packaging their modules properly, and they publish what they can on PyPI. The issue is, that PyPI (understandably) doesn't provide them with enough storage space to upload all versions of their module, and so they had to resort to publishing them themselves.

The unfortunate truth is, that hosting an html file with links is easier than an actual package index, and the PyTorch team should not be forced to host their own package index, even if only for their own packages, if PyPI simply refuses to support them - that is a dangerous precedent to be set, regardless of who is the maintainer of the affected package.

@Korijn
Copy link

Korijn commented Sep 16, 2020

@Korijn The pytorch team is packaging their modules properly, and they publish what they can on PyPI. The issue is, that PyPI (understandably) doesn't provide them with enough storage space to upload all versions of their module, and so they had to resort to publishing them themselves.

The unfortunate truth is, that hosting an html file with links is easier than an actual package index, and the PyTorch team should not be forced to host their own package index, even if only for their own packages, if PyPI simply refuses to support them - that is a dangerous precedent to be set, regardless of who is the maintainer of the affected package.

Fair enough, I have not digged deep enough to uncover this root cause. Apologies for my tone.

I do think that it's odd that hosting your own index is apparently too hard for such a project. Aren't there tons of such tools available to do that?

@golyalpha
Copy link

@Korijn I was saying that hosting an index is harder than hosting a static HTML file. Not in terms of setting it up (though it could be potentially harder to set up a package index than a static file host), but in terms of actually running it (expense, etc.)

@Korijn
Copy link

Korijn commented Sep 19, 2020

@Korijn I was saying that hosting an index is harder than hosting a static HTML file. Not in terms of setting it up (though it could be potentially harder to set up a package index than a static file host), but in terms of actually running it (expense, etc.)

Yes, well, I guess I was also not clear enough. One can also host a static HTML site that meets the PEP 503 format. That also allows it to be included as an index in pyproject.toml. See pytorch/pytorch#25639 and pytorch/pytorch#39779

@golyalpha
Copy link

Well, if the guys at PyTorch manage to figure it out, that's good. Poetry still probably should support the feature from pip.

@RafalSkolasinski
Copy link

RafalSkolasinski commented Sep 20, 2020

Hey guys, I read this thread (and the pytorch/pytorch#26340) and I just to make sure I didn't miss any workaround - do I understand it right that currently there is no way of including cpu version of torch into poetry's pyproject.toml?

@Korijn
Copy link

Korijn commented Sep 20, 2020

Hey guys, I read this thread (and the pytorch/pytorch#26340) and I just to make sure I didn't miss any workaround - do I understand it right that currently there is no way of including cpu version of torch into poetry's pyproject.toml?

If it's just torch (and not also vision), you can if you set up a PEP 503 compatible index with the package on it. There's a public one hacked together by @cpbotha you can use: https://vxlabs.com/pypi/

@sinoroc
Copy link

sinoroc commented Nov 17, 2020

I will add this as a potential workaround:

It does not seem to be a valid workaround, yet. It might be that there is an issue with URL redirections in poetry, so that the workaround fails for now.

@espdev
Copy link

espdev commented Nov 21, 2020

Any news on the PR?

For example, I need to use numpy+mkl on Windows 10 2004 Update because of this issue. I can download whl from gohlke web page only manually. And it would be nice to use --find-links and the local wheelhouse directory for searching wheel-packages instead of remote wheelhouse:

Something like this:

[tool.poetry.dependencies]
python = "^3.7"
numpy = { version = "^1.19.4+mkl", markers = "sys_platform == 'win32'" }
pandas = "^1.1.4"

[[tool.poetry.source]]
name = "numpy"
url = "wheels/"  # relative path to local directory with wheel files
find_links = true

@timothyjlaurent
Copy link

@Korijn's solution is good but it has become outdated.
https://eternalphane.github.io/pytorch-pypi/ seems to be up to date though.

@jhrmnn
Copy link
Contributor

jhrmnn commented May 21, 2021

Just linking to a related issue: #2613

@Nathan-Furnal
Copy link

Hi! Is there any news on this issue?

@Sbozzolo
Copy link

I stumbled upon this issue because vtk does not provide wheels for Python 3.10, but unofficial ones exist: pyvista/pyvista#2064

@ColdTeapot273K
Copy link

ColdTeapot273K commented May 3, 2022

Same problem when trying to install JAX via

pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_releases.html  # Note: wheels only available on linux.

Currently I've resorted to:

poetry run pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_releases.html    

Which effectively excludes JAX and its deps from tracking (i.e. they're not in poetry.lock)
Which in turn results in overwriting them with poetry-accessible dependencies once i try to install something that depends on them, i.e.:

> poetry add flax
...
Updating jaxlib (0.3.7+cuda11.cudnn82 -> 0.3.7) # "jax[cuda]" install gets overwritten effectively into "jax[cpu]"
...
Installing flax (0.4.1)

So i have to rerun the pip install command again after every poetry update. "Cat and mouse" basically 🐁🐈‍⬛ but at least it works.


UPD:
Found a community member has hosted a PEP 503 compliant index (similar to #1391 (comment)) for jax over here: jax-ml/jax#5410 (comment)

@mkniewallner
Copy link
Member

Hi, in to-be-released Poetry 1.2, it will be possible to define single page repositories following https://python-poetry.org/docs/master/repositories/#single-page-link-source.

You should be able to test that by using the recently released 1.2.0b2 pre-release.

@mkniewallner mkniewallner added status/waiting-on-response Waiting on response from author and removed hacktoberfest labels Jun 11, 2022
@mkniewallner
Copy link
Member

Closing, since it is now implemented in the next release of Poetry, 1.2, but if you still encounter any issue, feel free to reopen.

@mboisson
Copy link

--find-links accepts a directory path, not a single-page index. A single-page index supposes running a web server. This is not equivalent.

@rashidnhm
Copy link

We are running in an environment (out of control), where the packages are spread across different local directories. Pip was configured with --find-links to search across all the directories when installing a package, however, it does not seem like poetry is able to do the same.

Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/feature Feature requests/implementations
Projects
None yet
Development

No branches or pull requests