Add handling of multiple deps and different extras #3878

david-caro · 2016-07-27T15:19:47Z

Very simple handling of the case where you have multiple requirement
definitions that are the same, but differ on the extras, like:

mypgk[extra1]
mypgk[extra2]

will be merged into:

mypgk[extra1,extra2]

Signed-off-by: David Caro david@dcaro.es

piotr-dobrogost · 2016-07-27T15:41:01Z

Isn't this the same feature as in pull #3198?

david-caro · 2016-07-27T15:42:44Z

looks the same, looking (it did not work for me locally though... so if it is, it might be broken)

david-caro · 2016-07-27T15:44:17Z

it actually seems to be avoiding the duplicates, that is, ignoring the duplicated entry, instead of merging it into the existing one.

david-caro · 2016-07-27T15:48:05Z

hmmm... might it be that the if condition is wrong there? it actually aggregates the new requirement extras and the existing one, but the if says:

             if (parent_req_name is None and existing_req and not
                    existing_req.constraint and
                    existing_req.extras == install_req.extras):

so it will never get in if the extras are different.

david-caro · 2016-07-27T15:52:27Z

sorry... I'm wrong, that if is to throw the exception

david-caro · 2016-07-27T16:07:28Z

hmm... so the merging of the extras only gets done if the existing_req.constraint is false, and the install_req.constraint is true:

https://github.com/pypa/pip/blob/master/pip/req/req_set.py#L272

                if not install_req.constraint and existing_req.constraint:
                    if (install_req.link and not (existing_req.link and
                       install_req.link.path == existing_req.link.path)):
                        self.reqs_to_cleanup.append(install_req)
                        raise InstallationError(
                            "Could not satisfy constraints for '%s': "
                            "installation from path or url cannot be "
                            "constrained to a version" % name)
                    # If we're now installing a constraint, mark the existing
                    # object for real installation.
                    existing_req.constraint = False
                    existing_req.extras = tuple(
                        sorted(set(existing_req.extras).union(
                               set(install_req.extras))))
                    logger.debug("Setting %s extras to: %s",
                                 existing_req, existing_req.extras)
                    # And now we need to scan this.
                    result = [existing_req]

Maybe that's the if that is not ok? Shouldn't it be that none of their .constraint is True? (I'll try to find out what the constraint actually means though).

david-caro · 2016-07-27T16:18:53Z

A quick'n'dirty test just changing the if condition seems to work ok, will investigate but any info is really appreciated 😉

* Removes all the usage of extras for database related requirements and uses just the basic invenio-db module, to workaround a limitation of pip that is not yet solved, see pypa/pip#3878 for more info Signed-off-by: David Caro <david@dcaro.es>

david-caro · 2016-07-29T19:48:37Z

I think this is ready, added also a test for the special case.

* Removes all the usage of extras for database related requirements and uses just the basic invenio-db module, to workaround a limitation of pip that is not yet solved, see pypa/pip#3878 for more info Signed-off-by: David Caro <david@dcaro.es>

david-caro · 2016-08-15T15:02:01Z

ping?

david-caro · 2016-09-20T15:02:05Z

ping?

david-caro · 2016-10-18T11:38:13Z

ping?

* Removes all the usage of extras for database related requirements and uses just the basic invenio-db module, to workaround a limitation of pip that is not yet solved, see pypa/pip#3878 for more info Signed-off-by: David Caro <david@dcaro.es>

xavfernandez · 2016-10-26T21:37:59Z

This seems clean enough but this would (if I'm not mistakenà merge a pkg[extra1]<1 and a pkg[extra2]>=2 into a pkg[extra1,extra2]<1 (or >=2 depending on the resolution order).

This would not be much worth than pip's current behavior though.
Any opinion @pypa/pip-committers ?

david-caro · 2016-10-27T06:48:10Z

This pr already solves the issue of not doing the requirements merge at all, that already allows us to not have to run pip twice on deployment. Si I think that this already improves the current behavior.

Imo the handling of the versioning there might deserve another pull request.

david-caro · 2017-03-14T15:08:34Z

ping?

pradyunsg · 2017-11-04T05:24:09Z

tests/data/packages/LocalExtrasDuplicated/.gitignore

@@ -0,0 +1 @@
+/LocalExtras.egg-info


remove this?

pradyunsg · 2017-11-04T05:25:16Z

tests/data/packages/LocalExtrasDuplicated/setup.py

+    packages=find_packages(),
+    extras_require={ 'shrubbery': ['LocalExtras[baz]']},
+    install_requires=['LocalExtras[bar]'],
+    dependency_links=[DEP_URL]


You could just add simple as a dependency and use script.pip_install_local in the test, using the package names.

I reused the already existing LocalExtras package. So, to change that, here would be:

# No need for path_to_url, HERE, DEP_PATH or DEP_URL setup( ... install_requires=['simple', 'LocalExtras[Bar]'], ... )

And then in the test something like:

result = script.pip_local_install('--no-index', '-f', data.find_links, to_install + '[shrubbery]')

?

(it's been quite a long time, so my memory is a bit rusty)

Yes. Off the top of my head, the test would be:

result = script.pip_local_install(to_install + '[shrubbery]')

pradyunsg · 2017-11-04T05:26:31Z

pip/req/req_set.py

@@ -286,6 +286,14 @@ def add_requirement(self, install_req, parent_req_name=None):
                                 existing_req, existing_req.extras)
                    # And now we need to scan this.
                    result = [existing_req]
+                elif not install_req.constraint:


It'd be nice if you could rebase this; there's a few questions I wanna ask but I'll wait for a response from you before asking them.

pradyunsg · 2017-11-04T05:27:52Z

Hi @david-caro!

Sorry for the lack of a response for all this while; pip's very short on developer time.

If you could rebase or merge master in this PR and add a news file, that'd be nice. :)

david-caro · 2017-11-05T18:42:50Z

I've rebased over latest master, it seems to fail on travis for different reasons though, any help is appreciated (will try to check closer, but might take me some time).

pradyunsg · 2017-11-05T19:34:52Z

@david-caro I think I broke the master CI build with merge of #4739. Sorry! :|

I've a fix for the faulty test in #4841.

pradyunsg · 2017-11-05T19:38:05Z

NEWS.rst

@@ -18,6 +18,7 @@
 - Fix a crash on non ASCII characters from `lsb_release`. (#4062)
 - Fix an SyntaxError in an unused module of a vendored dependency. (#4059)
 - Fix UNC paths on Windows. (#4064)
+- Improve extras dependency handling (#3878)


We've had a change in process; so instead of modifying NEWS.rst, you'd want to add a file in news/ directory. More about that here: https://pip.pypa.io/en/latest/development/#adding-a-news-entry

pradyunsg · 2017-11-05T19:40:05Z

tests/data/packages/LocalExtrasDuplicated/setup.py

+    packages=find_packages(),
+    extras_require={ 'shrubbery': ['LocalExtras[baz]']},
+    install_requires=['LocalExtras[bar]'],
+    dependency_links=[DEP_URL]


Yes. Off the top of my head, the test would be:

result = script.pip_local_install(to_install + '[shrubbery]')

pradyunsg · 2017-11-06T07:46:13Z

@david-caro The tests are fixed on master. :)

Very simple handling of the case where you have multiple requirement definitions that are the same, but differ on the extras, like: mypgk[extra1] mypgk[extra2] defined in the setup.py file, either as `install_requires` or `extras_require`, now they will be merged into: mypgk[extra1,extra2] Instead of ignoring the latest appearances. Signed-off-by: David Caro <david@dcaro.es>

david-caro · 2017-11-24T16:50:25Z

Updated and now passing the tests too.

BrownTruck · 2018-03-21T08:30:07Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

pradyunsg · 2019-02-25T17:23:15Z

Gonna go ahead and close this due to lack of activity here.

lock · 2019-05-28T23:11:02Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

david-caro mentioned this pull request Jul 27, 2016

Resolve multiple extras requirements for repeated requires entries pypa/setuptools#696

Closed

david-caro changed the title ~~Add handling of multiple deps and different extras~~ WIP Add handling of multiple deps and different extras Jul 27, 2016

david-caro force-pushed the merge_multiple_extras branch from ee5dd85 to c7fcdfe Compare July 29, 2016 09:15

david-caro mentioned this pull request Jul 29, 2016

WIP tests: repeated package different extras inveniosoftware/requirements-builder#6

Closed

david-caro force-pushed the merge_multiple_extras branch 3 times, most recently from 8889b55 to 4898100 Compare July 29, 2016 18:44

david-caro changed the title ~~WIP Add handling of multiple deps and different extras~~ Add handling of multiple deps and different extras Jul 29, 2016

xavfernandez closed this Oct 26, 2016

xavfernandez reopened this Oct 26, 2016

piotr-dobrogost mentioned this pull request Nov 21, 2016

extras_require isn't honored if a package is already installed without them #3903

Closed

cjrh mentioned this pull request Mar 19, 2017

Issue #3: add SSL support nikipore/stompest#38

Merged

BrownTruck added the needs rebase or merge PR has conflicts with current master label Sep 1, 2017

pradyunsg reviewed Nov 4, 2017

View reviewed changes

david-caro force-pushed the merge_multiple_extras branch from 4898100 to 150f0d5 Compare November 5, 2017 18:23

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Nov 5, 2017

david-caro force-pushed the merge_multiple_extras branch from 150f0d5 to b5d22fd Compare November 5, 2017 18:41

pradyunsg reviewed Nov 5, 2017

View reviewed changes

pradyunsg added the C: extras Handling optional dependencies label Nov 11, 2017

pradyunsg closed this Nov 11, 2017

pradyunsg reopened this Nov 11, 2017

david-caro force-pushed the merge_multiple_extras branch from b5d22fd to 119c038 Compare November 12, 2017 15:50

david-caro force-pushed the merge_multiple_extras branch from 119c038 to 39a724c Compare November 12, 2017 16:22

BrownTruck added the needs rebase or merge PR has conflicts with current master label Mar 21, 2018

arthurdarcet mentioned this pull request Mar 27, 2018

Add optional dependencies, forwarding them to pymongo mongodb/motor#38

Closed

pradyunsg added the S: needs triage Issues/PRs that need to be triaged label May 11, 2018

pradyunsg mentioned this pull request Feb 25, 2019

Always merge extras from existing and incoming requirements #6242

Closed

pradyunsg added the C: dependency resolution About choosing which dependencies to install label Feb 25, 2019

pradyunsg closed this Feb 25, 2019

lock bot added the auto-locked Outdated issues that have been locked by automation label May 28, 2019

lock bot locked as resolved and limited conversation to collaborators May 28, 2019

pradyunsg removed the S: needs triage Issues/PRs that need to be triaged label Feb 21, 2020

pradyunsg removed the needs rebase or merge PR has conflicts with current master label Apr 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add handling of multiple deps and different extras #3878

Add handling of multiple deps and different extras #3878

david-caro commented Jul 27, 2016 •

edited by pradyunsg

Loading

piotr-dobrogost commented Jul 27, 2016

david-caro commented Jul 27, 2016

david-caro commented Jul 27, 2016

david-caro commented Jul 27, 2016

david-caro commented Jul 27, 2016

david-caro commented Jul 27, 2016 •

edited

Loading

david-caro commented Jul 27, 2016 •

edited

Loading

david-caro commented Jul 29, 2016

david-caro commented Aug 15, 2016

david-caro commented Sep 20, 2016

david-caro commented Oct 18, 2016

xavfernandez commented Oct 26, 2016

david-caro commented Oct 27, 2016 •

edited

Loading

david-caro commented Mar 14, 2017

pradyunsg Nov 4, 2017

pradyunsg Nov 4, 2017

david-caro Nov 5, 2017

pradyunsg Nov 5, 2017

pradyunsg Nov 4, 2017

pradyunsg commented Nov 4, 2017

david-caro commented Nov 5, 2017

pradyunsg commented Nov 5, 2017

pradyunsg Nov 5, 2017

pradyunsg Nov 5, 2017

pradyunsg commented Nov 6, 2017

david-caro commented Nov 24, 2017

BrownTruck commented Mar 21, 2018

pradyunsg commented Feb 25, 2019

lock bot commented May 28, 2019

		@@ -0,0 +1 @@
		/LocalExtras.egg-info

Add handling of multiple deps and different extras #3878

Add handling of multiple deps and different extras #3878

Conversation

david-caro commented Jul 27, 2016 • edited by pradyunsg Loading

piotr-dobrogost commented Jul 27, 2016

david-caro commented Jul 27, 2016

david-caro commented Jul 27, 2016

david-caro commented Jul 27, 2016

david-caro commented Jul 27, 2016

david-caro commented Jul 27, 2016 • edited Loading

david-caro commented Jul 27, 2016 • edited Loading

david-caro commented Jul 29, 2016

david-caro commented Aug 15, 2016

david-caro commented Sep 20, 2016

david-caro commented Oct 18, 2016

xavfernandez commented Oct 26, 2016

david-caro commented Oct 27, 2016 • edited Loading

david-caro commented Mar 14, 2017

pradyunsg Nov 4, 2017

Choose a reason for hiding this comment

pradyunsg Nov 4, 2017

Choose a reason for hiding this comment

david-caro Nov 5, 2017

Choose a reason for hiding this comment

pradyunsg Nov 5, 2017

Choose a reason for hiding this comment

pradyunsg Nov 4, 2017

Choose a reason for hiding this comment

pradyunsg commented Nov 4, 2017

david-caro commented Nov 5, 2017

pradyunsg commented Nov 5, 2017

pradyunsg Nov 5, 2017

Choose a reason for hiding this comment

pradyunsg Nov 5, 2017

Choose a reason for hiding this comment

pradyunsg commented Nov 6, 2017

david-caro commented Nov 24, 2017

BrownTruck commented Mar 21, 2018

pradyunsg commented Feb 25, 2019

lock bot commented May 28, 2019

david-caro commented Jul 27, 2016 •

edited by pradyunsg

Loading

david-caro commented Jul 27, 2016 •

edited

Loading

david-caro commented Jul 27, 2016 •

edited

Loading

david-caro commented Oct 27, 2016 •

edited

Loading