Support URLs as packages #807

jcushman · 2019-05-03T20:07:30Z

This PR handles URLs in requirements.in, building on work by @nim65s and @toejough.

This is similar to the previous PR #764, but I rebased @toejough's latest on top of master, updated the PyPI download url and some tests, and added support for --generate-hashes.

Note that URL requirements don't work for packages that use setuptools_scm, including pip-tools itself, but that's an inherent limitation of URL requirements and not an issue with pip-compile.

Changelog-friendly one-liner: Support URLs as packages

Contributor checklist

Provided the tests for the changes.
Requested a review from another contributor.
Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

codecov · 2019-05-03T20:16:45Z

Codecov Report

Merging #807 into master will increase coverage by 0.01%.
The diff coverage is 97.05%.

@@            Coverage Diff             @@
##           master     #807      +/-   ##
==========================================
+ Coverage   98.82%   98.83%   +0.01%     
==========================================
  Files          36       36              
  Lines        2119     2147      +28     
  Branches      275      278       +3     
==========================================
+ Hits         2094     2122      +28     
  Misses         15       15              
  Partials       10       10

Impacted Files	Coverage Δ
piptools/repositories/base.py	`100% <ø> (ø)`	⬆️
tests/test_resolver.py	`100% <ø> (ø)`	⬆️
piptools/exceptions.py	`86.11% <ø> (-2.27%)`	⬇️
piptools/resolver.py	`100% <100%> (ø)`	⬆️
piptools/utils.py	`100% <100%> (ø)`	⬆️
piptools/scripts/compile.py	`100% <100%> (ø)`	⬆️
piptools/writer.py	`100% <100%> (ø)`	⬆️
tests/test_utils.py	`100% <100%> (ø)`	⬆️
tests/test_cli_compile.py	`100% <100%> (ø)`	⬆️
tests/test_sync.py	`100% <100%> (ø)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 990a3d0...de874f4. Read the comment docs.

auvipy

it would be great if you could increase coverage

atugushev

So many tries to implement this feature, hope this will be eventually merged. Thank you for great work! Have minor comments. And it would be nice not to decrease the test coverage.

piptools/utils.py

piptools/repositories/pypi.py

piptools/utils.py

tests/test_cli_compile.py

piptools/utils.py

blueyed · 2019-05-04T10:17:01Z

Thanks for working on this!

Tried this quickly, and it fails when not using #egg=foo at the end:

Current constraints:
  git+https://github.com/blueyed/starlette.git@local

Finding the best candidates:
  found candidate git+https://github.com/blueyed/starlette.git@local (constraint was <any>)

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

None
Traceback (most recent call last):
  File "…/Vcs/pip-tools/.venv/bin/pip-compile", line 11, in <module>
    load_entry_point('pip-tools', 'console_scripts', 'pip-compile')()
  File "…/Vcs/pip-tools/.venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "…/Vcs/pip-tools/.venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "…/Vcs/pip-tools/.venv/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "…/Vcs/pip-tools/.venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "…/Vcs/pip-tools/.venv/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "…/Vcs/pip-tools/piptools/scripts/compile.py", line 415, in cli
    key_from_req(ireq.req) for ireq in constraints if not ireq.constraint
  File "…/Vcs/pip-tools/piptools/scripts/compile.py", line 415, in <setcomp>
    key_from_req(ireq.req) for ireq in constraints if not ireq.constraint
  File "…/Vcs/pip-tools/piptools/utils.py", line 30, in key_from_req
    key = req.name
AttributeError: 'NoneType' object has no attribute 'name'

Used git+https://github.com/blueyed/starlette.git@local - with git+https://github.com/blueyed/starlette.git@local#egg=starlette it worked, but just left it as-is - is that expected?

jcushman · 2019-05-05T02:29:53Z

Thanks for the reviews! I believe my updates address all the comments so far. I was able to greatly simplify the is_url_requirement function, which was the messiest part of the logic.

@blueyed:

it fails when not using #egg=foo at the end

Good catch -- this should be fixed now.

with git+https://github.com/blueyed/starlette.git@local#egg=starlette it worked, but just left it as-is - is that expected?

I think so, yes -- it should just copy URLs directly to requirements.txt, along with their deps. Looks like starlette doesn't have any dependencies so you'd just see the original URL.

blueyed · 2019-05-05T02:48:12Z

it should just copy URLs directly to requirements.txt

Shouldn't it pin it at the current commit? (that's what I did in #169 back then)

jcushman · 2019-05-05T16:18:49Z

Update: the code I was adapting only applied to http/scm URLs and excluded file URLs, but I think file URLs actually work just fine. The latest commit removes the exclusion and adds tests for file URLs. This also lets us drop the whole UnsupportedConstraint codepath because we're no longer excluding anything as unsupported.

updated

auvipy

can we skip the pip master failures and merged this as is?

atugushev

I've noticed that pip-sync would try to sync URL package every time.

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
https://github.com/pallets/click/archive/7.0.zip

$ pip-sync
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting https://github.com/pallets/click/archive/7.0.zip (from -r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmp7O5s_H (line 1))
  Downloading https://github.com/pallets/click/archive/7.0.zip
     - 1.3MB 2.6MB/s
Building wheels for collected packages: Click
  Building wheel for Click (setup.py) ... done
  Stored in directory: /private/var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/pip-ephem-wheel-cache-1eHUT2/wheels/3f/4c/b1/bb74f5eeb10363c51faaaa45211c996886de6beb3fbd804067
Successfully built Click
Installing collected packages: Click
  Found existing installation: click 6.7
    Uninstalling click-6.7:
      Successfully uninstalled click-6.7
Successfully installed Click-7.0
 
$ pip-sync
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting https://github.com/pallets/click/archive/7.0.zip (from -r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpW22iN9 (line 1))
  Downloading https://github.com/pallets/click/archive/7.0.zip
     - 624kB 2.2MB/s
Requirement already satisfied (use --upgrade to upgrade): Click==7.0 from https://github.com/pallets/click/archive/7.0.zip in ./.venv/lib/python2.7/site-packages (from -r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpW22iN9 (line 1))
Building wheels for collected packages: Click
  Building wheel for Click (setup.py) ... done
  Stored in directory: /private/var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/pip-ephem-wheel-cache-oY4xDG/wheels/3f/4c/b1/bb74f5eeb10363c51faaaa45211c996886de6beb3fbd804067
Successfully built Click

On the second pip-sync it's only built wheels and warned that Requirement already satisfied.

jcushman · 2019-05-05T17:17:24Z

Shouldn't it pin it at the current commit? (that's what I did in #169 back then)

Oh, I see what you mean! Yes, it would be better to pin vcs URLs at the current commit.

What do you think of saving that feature for a separate pull request? The current PR is not a regression, AFAIK -- it treats git+https://github.com/blueyed/starlette.git@local#egg=starlette the same as -e git+https://github.com/blueyed/starlette.git@local#egg=starlette, which is also not pinned. In the meantime, you can work around the limitation (as I already do with -e) by using a commit URL instead of a tag/branch.

jcushman · 2019-05-05T18:05:38Z

I've noticed that pip-sync would try to sync URL package every time.

Good catch. The latest commit allows you to avoid this by providing the pinned version in the egg fragment:

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
https://github.com/pallets/click/archive/7.0.zip#egg=click==7.0
$ pip-sync
Everything up-to-date

I think that's the best we can do, right? If the version isn't provided in the URL, the archive has to be downloaded and checked.

atugushev · 2019-05-05T23:58:24Z

Good catch. The latest commit allows you to avoid this by providing the pinned version in the egg fragment:

Great!

I think that's the best we can do, right? If the version isn't provided in the URL, the archive has to be downloaded and checked.

You are right. Turns out that's correct behaviour.

BTW, i've tried egg fragment with only package name:

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
https://github.com/pallets/click/archive/7.0.zip#egg=click

$ pip-sync
Everything up-to-date

Shouldn't it download and check version if there is only package name in egg fragment and no version?

jcushman · 2019-05-06T02:55:20Z

Shouldn't it download and check version if there is only package name in egg fragment and no version?

Oops, yes it should. In fact it should download and reinstall, even if the version number matches in the download, because we don't know if the URL has changed. For example, suppose you change requirements.in from:

https://github.com/jazzband/pip-tools/archive/master.zip#egg=example

to

https://github.com/jcushman/pip-tools/archive/master.zip#egg=example

It would be surprising if pip-sync didn't change anything because I didn't update the version in my fork, right? We don't know how the installed module was specified, so we have to uninstall and reinstall on every pip-sync.

My latest commit hopefully strikes a good balance:

If you choose to explicitly provide #egg=example==7.0 in requirements.txt, then pip-sync believes you and only downloads and reinstalls if the version doesn't match.
If you just provide #egg=example, or no name, then pip-sync downloads, uninstalls, and reinstalls.

atugushev · 2019-05-06T04:11:55Z

My latest commit hopefully strikes a good balance:

If you choose to explicitly provide #egg=example==7.0 in requirements.txt, then pip-sync believes you and only downloads and reinstalls if the version doesn't match.

If you just provide #egg=example, or no name, then pip-sync downloads, uninstalls, and reinstalls.

Awesome! Would've been nice to have tests for each scenario in test_cli_sync.py.

jcushman · 2019-05-06T14:01:47Z

Would've been nice to have tests for each scenario in test_cli_sync.py.

Those scenarios are tested via diff() unit tests in test_sync.py, the same as the rest of the pip-sync diff behavior. test_cli_sync.py only seems to have tests for cli wrapper flags and error messages.

What else is left to get this merged? I looked at the codecov/patch report, and I think the only lines that lost coverage are guards that are no longer hit, but that I'm wary of removing in case there are edge cases I don't know about.

jcushman · 2019-05-06T19:04:25Z

Squashed! git rebase -i HEAD~19 is a personal record for me. :)

jcushman · 2019-05-08T13:06:15Z

Anything else pending on this?

jedie · 2019-05-09T06:23:35Z

Anything else pending on this?

Seems that this will land in 3.7.0 and there is one issued open, see: https://github.com/jazzband/pip-tools/milestone/19 isn't it?

atugushev · 2019-05-09T06:52:12Z

Seems that this will land in 3.7.0 and there is one issued open, see: https://github.com/jazzband/pip-tools/milestone/19 isn't it?

We can postpone this issue to the next release.

atugushev · 2019-05-09T06:57:02Z

I'll prepare a release within next 24 hours.

vphilippon · 2019-05-09T17:08:49Z

@atugushev I validated pip-tools 3.7.0 release, it's on pypi.org now

See jazzband/pip-tools#807 for more information about permitting URLs as packages. We want this because --generate-hashes works with URL-based requirements but not with Git-repo-based requirements. Subtle different, but it works. As such I think it's okay to remove the separate github URL requirements file. One other thing I do here is reconcile the pshtt/ssylze/cryptography dependency-hell situation by pinning the version of sslyze to the latest one that supports (a) the most recent, secure version of cryptography, and (b) python 3.5. Some of the versions of these packages were not release the last time this situation was examined.

See jazzband/pip-tools#807 for more information about permitting URLs as packages. We want this because --generate-hashes works with URL-based requirements but not with Git-repo-based requirements. Subtle different, but it works. As such I think it's okay to remove the separate github URL requirements file.

jcushman force-pushed the url-requirements branch from 1fb3569 to c33861b Compare May 3, 2019 20:20

jcushman requested review from atugushev, blueyed and vphilippon May 3, 2019 20:46

auvipy reviewed May 4, 2019

View reviewed changes

atugushev previously requested changes May 4, 2019

View reviewed changes

atugushev added enhancement Improvements to functionality non-editable VCS/URL labels May 4, 2019

auvipy added this to the 3.7.0 milestone May 4, 2019

jcushman force-pushed the url-requirements branch 2 times, most recently from 5d59167 to 3263535 Compare May 5, 2019 02:03

jcushman requested a review from atugushev May 5, 2019 02:30

jcushman force-pushed the url-requirements branch from ce26f39 to 8e68a18 Compare May 5, 2019 16:12

auvipy reviewed May 5, 2019

View reviewed changes

atugushev reviewed May 5, 2019

View reviewed changes

jcushman force-pushed the url-requirements branch from b51b212 to 225ebb9 Compare May 5, 2019 18:01

jcushman force-pushed the url-requirements branch 2 times, most recently from 393d434 to 6129847 Compare May 6, 2019 02:53

support URLs as packages

de874f4

jcushman force-pushed the url-requirements branch from 6129847 to de874f4 Compare May 6, 2019 18:43

jazzband deleted a comment from codecov bot May 6, 2019

auvipy approved these changes May 6, 2019

View reviewed changes

auvipy merged commit 4d6d355 into jazzband:master May 8, 2019

This was referenced May 8, 2019

support URLs as packages, fix #272 #574

Closed

Remove the restriction on package urls because it is unnecessary #405

Closed

--generate-hashes with remote URLs #700

Closed

pip-compile: handle zip/tar.gz requirements #93

Closed

jcushman mentioned this pull request May 10, 2019

Fix --generate-hashes with bare VCS URLs #812

Merged

4 tasks

atugushev mentioned this pull request May 16, 2019

support pinned vcs dependencies [#355] #372

Closed

jmbowman mentioned this pull request May 21, 2019

Add support for non-editable vcs requirements #355

Closed

timabbott mentioned this pull request May 21, 2019

Clean up update-locked-requirements and requirements.in files to remove -e hackery. zulip/zulip#12374

Closed

nickpell mentioned this pull request Jun 10, 2019

Remove -e from repo requirements dimagi/commcare-hq#24540

Merged

atugushev mentioned this pull request Jul 23, 2019

3.9.0 pip-compile gets confused by GitHub installs without -e #851

Closed

atugushev mentioned this pull request Sep 15, 2019

Unable to Put a Wheel in requirements.in File #415

Closed

awbirdsall mentioned this pull request Oct 15, 2019

404 error using github hashable URL #947

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support URLs as packages #807

Support URLs as packages #807

jcushman commented May 3, 2019 •

edited by atugushev

Loading

codecov bot commented May 3, 2019 •

edited

Loading

auvipy left a comment

atugushev left a comment

blueyed commented May 4, 2019

jcushman commented May 5, 2019

blueyed commented May 5, 2019

jcushman commented May 5, 2019 •

edited

Loading

auvipy left a comment

atugushev left a comment

jcushman commented May 5, 2019

jcushman commented May 5, 2019

atugushev commented May 5, 2019

jcushman commented May 6, 2019

atugushev commented May 6, 2019

jcushman commented May 6, 2019

jcushman commented May 6, 2019

jcushman commented May 8, 2019

jedie commented May 9, 2019

atugushev commented May 9, 2019

atugushev commented May 9, 2019

vphilippon commented May 9, 2019

Support URLs as packages #807

Support URLs as packages #807

Conversation

jcushman commented May 3, 2019 • edited by atugushev Loading

Contributor checklist

codecov bot commented May 3, 2019 • edited Loading

Codecov Report

auvipy left a comment

Choose a reason for hiding this comment

atugushev left a comment

Choose a reason for hiding this comment

blueyed commented May 4, 2019

jcushman commented May 5, 2019

blueyed commented May 5, 2019

jcushman commented May 5, 2019 • edited Loading

auvipy left a comment

Choose a reason for hiding this comment

atugushev left a comment

Choose a reason for hiding this comment

jcushman commented May 5, 2019

jcushman commented May 5, 2019

atugushev commented May 5, 2019

jcushman commented May 6, 2019

atugushev commented May 6, 2019

jcushman commented May 6, 2019

jcushman commented May 6, 2019

jcushman commented May 8, 2019

jedie commented May 9, 2019

atugushev commented May 9, 2019

atugushev commented May 9, 2019

vphilippon commented May 9, 2019

jcushman commented May 3, 2019 •

edited by atugushev

Loading

codecov bot commented May 3, 2019 •

edited

Loading

jcushman commented May 5, 2019 •

edited

Loading