Multiple entries in requirements.in with the same file cause collision

[geokala]

When a requirements.in contains multiple entries with the same file name, there is a collision when they are downloaded. e.g.

requirements.in contents:
https://github.com/cloudify-cosmo/cloudify-common/archive/master.zip#egg=cloudify-common[dispatcher]
https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip#egg=cloudify

This results in a warning that the cloudify 5.1.0.dev1 does not provide the extra 'dispatcher',
followed by an error that no version could be found for cloudify-common==5.1.0.dev1

Environment Versions

1. Ubuntu 18.04
2. Python version: Python 2.7.17 # Though by the behaviour I expect this to still cause problems when we switch this to Python 3.
3. pip version: pip 20.1
4. pip-tools version: pip-compile, version 5.1.2

Steps to replicate

1. Create an empty dir, and cd to it
2. Make a requirements.in file in this dir, containing:
   https://github.com/cloudify-cosmo/cloudify-common/archive/master.zip#egg=cloudify-common[dispatcher]
https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip#egg=cloudify
3. Run pip-compile

Expected result

requirements.txt is produced.

Actual result

Errors abound (see above for specific errors).

[atugushev]

Hello @geokala,

Thanks for the issue! This problem is not related to how many entries with the same filenames in requirements.in. Actually, the issue is with setup.py in cloudify-cli. There cloudify-common pinned to 5.1.0.dev1, which is not uploaded to PyPI. See the reproducer and error message:

$ cat requirements.in
https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip#egg=cloudify

$ pip-compile

ERROR: Could not find a version that satisfies the requirement cloudify-common==5.1.0.dev1 (from cloudify->-r requirements.in (line 2)) (from versions: 4.4, 4.5, 4.5.5, 4.6, 5.0.0, 5.0.5.1)

I'd suggest to open issue on cloudify-cli's issue tracker.

[geokala]

Hi @atugushev
There is indeed a pinning of that version, but that is provided in the first line.

The dependencies work just fine if you put them in a requirements.txt file and pip install -r requirements.txt

They're failing to work when used with pip-compile (possibly due to one of the tools being used by it under the hood), probably because there's a cached master.zip being overwritten (I think- from discussion with a coworker- haven't checked directly myself).

Based on that, I don't believe this bug is invalid.

(I'm actually working on cloudify and we're using this for some pre-release testing.)

Thanks,
S

[atugushev]

Yes, indeed. Sorry, I've overlooked that fact. I've compiled with --verbose flag and noticed the warning:

Finding secondary dependencies:
  Collecting cloudify
    Using cached https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip
    Saved /root/.cache/pip-tools/pkgs/master.zip
  Collecting cloudify-common
    File was already downloaded /root/.cache/pip-tools/pkgs/master.zip
    WARNING: Generating metadata for package cloudify-common produced metadata for project name cloudify. Fix your #egg=cloudify-common fragments.

There must've been a bug with conflicted cache names. I made an experiment and it's confirmed your suspicious:

$ cd /tmp

$ wget https://github.com/cloudify-cosmo/cloudify-common/archive/master.zip -O cloudify-common-master.zip

$ wget https://github.com/cloudify-cosmo/cloudify-cli/archive/master.zip -O cloudify-cli-master.zip

$ cat requirements.in
file:///tmp/cloudify-common-master.zip#egg=cloudify-common[dispatcher]
file:///tmp/cloudify-cli-master.zip#egg=cloudify

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
appdirs==1.4.4            # via virtualenv
backports.shutil_get_terminal_size==1.0.0  # via cloudify
bottle==0.12.18           # via cloudify-common
certifi==2020.4.5.1       # via requests
chardet==3.0.4            # via requests
click==7.1.2              # via click-didyoumean, cloudify
click_didyoumean==0.0.3   # via cloudify
file:///tmp/cloudify-cli-master.zip#egg=cloudify  # via -r requirements.in
file:///tmp/cloudify-common-master.zip#egg=cloudify-common[dispatcher]  # via -r requirements.in, cloudify
colorama==0.3.3           # via cloudify
configparser==4.0.2       # via importlib-metadata
contextlib2==0.6.0.post1  # via importlib-metadata, importlib-resources, virtualenv
decorator==4.4.2          # via networkx
distlib==0.3.0            # via virtualenv
filelock==3.0.12          # via virtualenv
idna==2.9                 # via requests
importlib-metadata==1.6.0  # via importlib-resources, virtualenv
importlib-resources==1.5.0  # via virtualenv
ipaddress==1.0.19         # via cloudify
jinja2==2.10              # via cloudify, cloudify-common
markupsafe==1.1.1         # via jinja2
networkx==1.9.1           # via cloudify-common
pathlib2==2.3.5           # via importlib-metadata, importlib-resources, virtualenv
pika==0.11.2              # via cloudify-common
ply==3.11                 # via pysmi
prettytable==0.7.2        # via cloudify
proxy_tools==0.1.0        # via cloudify-common
pyasn1==0.4.8             # via pysnmp
pycryptodomex==3.9.7      # via pysnmp
pysmi==0.3.4              # via pysnmp
pysnmp==4.4.5             # via cloudify-common
pyyaml==3.12              # via cloudify, cloudify-common
requests==2.23.0          # via cloudify, cloudify-common, requests-toolbelt
requests_toolbelt==0.8.0  # via cloudify-common
retrying==1.3.3           # via cloudify, cloudify-common
scandir==1.10.0           # via pathlib2
singledispatch==3.4.0.3   # via importlib-resources
six==1.14.0               # via pathlib2, retrying, virtualenv
typing==3.7.4.1           # via importlib-resources
urllib3==1.25.9           # via requests
virtualenv==20.0.20       # via wagon
wagon[venv]==0.6.3        # via cloudify
wheel==0.29.0             # via wagon
zipp==1.2.0               # via importlib-metadata, importlib-resources

# The following packages are considered to be unsafe in a requirements file:
# setuptools

Thanks for the good catch @geokala! 👍

[geokala]

No worries.

Do you have any rough idea of where the root cause might be? I might (depending on other ongoing stuff) try to take a dig to fix it myself.

[atugushev]

Sure! This download dir is global for all links:

pip-tools/piptools/repositories/pypi.py

Line 86 in b1d7369

self._download_dir = fs_str(os.path.join(self._cache_dir, "pkgs"))

There pip-tools passes this dir to pip:

pip-tools/piptools/repositories/pypi.py

Line 161 in b1d7369

download_dir=download_dir,

The idea might be to add to the download directory (before it passed to make_requirement_preparer) a salt based on a link URL to avoid collision, for example:

salt = hashlib.md5(ireq.link.url_without_fragment.encode()).hexdigest()
donwload_dir=os.path.join(download_dir, salt[:2], salt[2:4], salt[4:6], ...)

[geokala]

Sounds good, I'll try to have a look at it one evening this week!

[geokala]

OK, basic fix seems to work, I'm just looking at making a test before I PR it.

[geokala]

PR is up: #1149
Let me know if there are any issues with it.

[atugushev]

pip-tools v5.2.0 is released 🚀