Possible bug: problems resolving pytorch (cuda).

[fralik]

This may very well be PyTorch issue due to the way they distribute packages, but I will describe it here too.
The problem is that pip-compile doesn't stick to the way how package version is descried in .in file.
I am trying to install the default version of PyTorch (which comes compiled against CUDA 10.1).

Environment Versions

1. OS Type: Windows 10
2. Python version: 3.7.6
3. pip version: 20.0.2
4. pip-tools version: 4.5.1

Steps to replicate

1. Make requirements.in file:

   torch===1.4.0
   torchvision===0.5.0
   

   Note usage of arbitrary equality. I know that this is discouraged by pip. This is way I think the issue may be more related to PyTorch itself.
2. Compile it pip-compile -f https://download.pytorch.org/whl/torch_stable.html -o requirements.txt requirements.in

Expected result

Expected requirements.txt with torch===1.4.0 (note triple =).

Actual result

Actual result it:

torch==1.4.0              # via -r req.in, torchvision
torchvision===0.5.0       # via -r req.in

Note double = on torch and triple = on torchvision.

If one installs the packages via pip-sync requirements.txt, then:

Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting numpy==1.18.3
  Using cached numpy-1.18.3-cp37-cp37m-win_amd64.whl (12.8 MB)
Collecting pillow==7.1.1
  Using cached Pillow-7.1.1-cp37-cp37m-win_amd64.whl (2.0 MB)
Collecting torch==1.4.0
  Using cached https://download.pytorch.org/whl/cu92/torch-1.4.0%2Bcu92-cp37-cp37m-win_amd64.whl (641.8 MB)
Collecting torchvision===0.5.0
  Using cached torchvision-0.5.0-cp37-cp37m-win_amd64.whl (1.2 MB)

See that cu92 version of torch is selected.

However, if requirements.txt would contain torch===1.4.0, then the results of pip-sync would be:

Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch===1.4.0
  Using cached https://download.pytorch.org/whl/cu101/torch-1.4.0-cp37-cp37m-win_amd64.whl (796.8 MB)
Collecting torchvision===0.5.0
  Using cached torchvision-0.5.0-cp37-cp37m-win_amd64.whl (1.2 MB)
Successfully installed torch-1.4.0 torchvision-0.5.0

[atugushev]

Hello @fralik,

Thanks for the issue! Semantically === and == are equal. TBH I have no idea why pip chooses different wheels depending on version matcher specifier. However, I cannot reproduce on macOS, see:

With ==

$ cat requirements.txt
torch==1.4.0

$ pip-sync
Collecting torch==1.4.0
  Downloading torch-1.4.0-cp38-none-macosx_10_9_x86_64.whl (81.2 MB)

With ===

$ cat requirements.txt
torch==1.4.0

$ pip-sync
Collecting torch===1.4.0
  Downloading torch-1.4.0-cp38-none-macosx_10_9_x86_64.whl (81.2 MB)

UPDATE: ah, sorry, I should use --find-links=https://download.pytorch.org/whl/torch_stable.html, but it looks like it doesn't reproduce because of there are no cpu wheels for macOS.

UPDATE 2: can confirm on Linux with docker.

[atugushev]

A stacktrace for curious ones:

user@home:/$ docker run --rm -it python:3.8-slim /bin/bash
root@f8514dadee50:/# pip install torch==1.4.0 --find-links https://download.pytorch.org/whl/torch_stable.html

[0]   /usr/local/bin/pip(8)<module>()
-> sys.exit(main())
[1]   /usr/local/lib/python3.8/site-packages/pip/_internal/cli/main.py(75)main()
-> return command.main(cmd_args)
[2]   /usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py(105)main()
-> return self._main(args)
[3]   /usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py(186)_main()
-> status = self.run(options, args)
[4]   /usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py(331)run()
-> resolver.resolve(requirement_set)
[5]   /usr/local/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py(177)resolve()
-> discovered_reqs.extend(self._resolve_one(requirement_set, req))
[6]   /usr/local/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py(333)_resolve_one()
-> abstract_dist = self._get_abstract_dist_for(req_to_install)
[7]   /usr/local/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py(281)_get_abstract_dist_for()
-> req.populate_link(self.finder, upgrade_allowed, require_hashes)
[8]   /usr/local/lib/python3.8/site-packages/pip/_internal/req/req_install.py(249)populate_link()
-> self.link = finder.find_requirement(self, upgrade)
[9]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(898)find_requirement()
-> best_candidate_result = self.find_best_candidate(
[10]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(887)find_best_candidate()
-> return candidate_evaluator.compute_best_candidate(candidates)
[11]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(584)compute_best_candidate()
-> applicable_candidates = self.get_applicable_candidates(candidates)
[12]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(459)get_applicable_candidates()
-> versions = {
[13]   /usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py(459)<setcomp>()
-> versions = {
[14]   /usr/local/lib/python3.8/site-packages/pip/_vendor/packaging/specifiers.py(237)filter()
-> if self.contains(parsed_version, **kw):
[15]   /usr/local/lib/python3.8/site-packages/pip/_vendor/packaging/specifiers.py(221)contains()
-> return operator_callable(normalized_item, version)
[16]   /usr/local/lib/python3.8/site-packages/pip/_vendor/packaging/specifiers.py(324)wrapped()
-> return fn(self, prospective, spec)
[17] > /usr/local/lib/python3.8/site-packages/pip/_vendor/packaging/specifiers.py(503)_compare_equal()
-> prospective = Version(prospective.public)
(Pdb++)

>>> print(self)
<Specifier('==1.4.0')>

>>> print(prospective)
<Version('1.4.0+cpu')>

>>> print(prospective.public)
'1.4.0'

[atugushev]

PEP-440 about arbitrary === operator https://www.python.org/dev/peps/pep-0440/#arbitrary-equality

[atugushev]

Could be related also to #785, since scm dev versions have the same notation like 1.0.1.dev3+gd7cef3b.

[IceTDrinker]

Any news on this ? I have the same issue where I need to force the torch version with === and the generated .txt file only contains == which does not seem to enforce the proper version when using pip-sync

Cheers

[atugushev]

Hello @IceTDrinker,

Thanks for pinging. AFAIK, nobody works on the fix. Any help is highly appreciated and welcomed!

[IceTDrinker]

A pointer as to where I should look ? 🙂

[atugushev]

@IceTDrinker

Probably, this could be a good start:

pip-tools/piptools/utils.py

Line 78 in f60fb00

str(f"{name}{extras_string}=={version}"), constraint=constraint

Note hardcoded ==. Perhaps, there should be some way to resolve === to ===.

[IceTDrinker]

Looks like a good place to start indeed. I'll probably take a look this week-end thanks !