Cyclic dependencies found in Products.CMFPlone

[ericof]

BUG/PROBLEM REPORT

Installing Plone with pip (for newer versions of pip) is not possible due to Cyclic dependencies between Products.CMFPlone and some of its dependencies.

This affects Plone 5.2x and above.

What I did:

With a pip version greater than 20.3

python -m venv . 
./bin/pip install -U pip setuptools wheel 
./bin/pip install Plone Paste -c https://dist.plone.org/release/5.2.4/constraints.txt

What I expect to happen:

Plone to be installed in the virtual environment

What actually happened:

pip never finishes the installation.

Additional information

After installing pipdeptree and running it against a Plone installation with pip < 20.3, we receive the following warning (and then the script enters an infinite loop).

Warning!! Cyclic dependencies found
* Products.CMFPlone => plone.app.content => Products.CMFPlone
* Products.CMFPlone => plone.app.contentlisting => Products.CMFPlone
* Products.CMFPlone => plone.app.contentmenu => Products.CMFPlone
* Products.CMFPlone => plone.app.contentrules => Products.CMFPlone
* Products.CMFPlone => plone.app.contenttypes => Products.CMFPlone
* Products.CMFPlone => plone.app.dexterity => Products.CMFPlone
* Products.CMFPlone => plone.app.discussion => Products.CMFPlone
* Products.CMFPlone => plone.app.layout => Products.CMFPlone
* plone.app.portlets => plone.app.layout => plone.app.portlets
* Products.CMFPlone => plone.app.multilingual => Products.CMFPlone
* Products.CMFPlone => plone.app.portlets => Products.CMFPlone
* plone.app.layout => plone.app.portlets => plone.app.layout
* plone.app.vocabularies => plone.app.querystring => plone.app.vocabularies
* Products.CMFPlone => plone.app.registry => Products.CMFPlone
* Products.CMFPlone => plone.app.theming => Products.CMFPlone
* Products.CMFPlone => plone.app.users => Products.CMFPlone
* plone.app.querystring => plone.app.vocabularies => plone.app.querystring
* plone.app.z3cform => plone.app.widgets => plone.app.z3cform
* Products.CMFPlone => plone.app.workflow => Products.CMFPlone
* plone.app.widgets => plone.app.z3cform => plone.app.widgets
* plone.app.workflow => Products.CMFPlone => plone.app.workflow
* plone.app.users => Products.CMFPlone => plone.app.users
* plone.app.theming => Products.CMFPlone => plone.app.theming
* plone.app.registry => Products.CMFPlone => plone.app.registry
* plone.app.portlets => Products.CMFPlone => plone.app.portlets
* plone.app.multilingual => Products.CMFPlone => plone.app.multilingual
* plone.app.layout => Products.CMFPlone => plone.app.layout
* plone.app.discussion => Products.CMFPlone => plone.app.discussion
* plone.app.dexterity => Products.CMFPlone => plone.app.dexterity
* plone.app.contenttypes => Products.CMFPlone => plone.app.contenttypes
* plone.app.contentrules => Products.CMFPlone => plone.app.contentrules
* plone.app.contentmenu => Products.CMFPlone => plone.app.contentmenu
* plone.app.contentlisting => Products.CMFPlone => plone.app.contentlisting
* plone.app.content => Products.CMFPlone => plone.app.content

What version of Plone/ Addons I am using:

[ericof]

Dependencies

Products.CMFPlone

• plone.app.content (Remove cyclic dependency with Products.CMFPlone. plone.app.content#233)
• plone.app.contentlisting (Remove cyclic dependency with Products.CMFPlone. plone.app.contentlisting#38)
• plone.app.contentmenu (Remove cyclic dependency with Products.CMFPlone. plone.app.contentmenu#32)
• plone.app.contentrules (Remove cyclic dependency with Products.CMFPlone. plone.app.contentrules#65)
• plone.app.contenttypes (Remove cyclic dependency with Products.CMFPlone. plone.app.contenttypes#610)
• plone.app.dexterity (Remove cyclic dependency with Products.CMFPlone. plone.app.dexterity#330)
• plone.app.discussion (Remove cyclic dependency with Products.CMFPlone. plone.app.discussion#187)
• plone.app.layout (Remove cyclic dependency with Products.CMFPlone. plone.app.layout#262)
• plone.app.multilingual (Remove cyclic dependency with Products.CMFPlone. plone.app.multilingual#392)
• plone.app.portlets (Remove cyclic dependency with Products.CMFPlone and plone.app.layout plone.app.portlets#153)
• plone.app.registry (Remove cyclic dependency with Products.CMFPlone plone.app.registry#55)
• plone.app.theming (Remove cyclic dependency with Products.CMFPlone plone.app.theming#202)
• plone.app.users (Remove cyclic dependency with Products.CMFPlone plone.app.users#103)
• plone.app.workflow (Remove cyclic dependency with Products.CMFPlone plone.app.workflow#31)

plone.app.z3cform

• plone.app.widgets (Remove cyclic dependency with plone.app.z3cform plone.app.widgets#212)

plone.app.vocabularies

• plone.app.querystring (Remove cyclic dependency with Products.CMFPlone and plone.app.vocabul… plone.app.querystring#102)

plone.app.layout

• plone.app.portlets (Remove cyclic dependency with Products.CMFPlone and plone.app.layout plone.app.portlets#153)

[mauritsvanrees]

All merged. Thanks a lot @ericof and @sneridagh !

Now let's see if Jenkins stays green. Then I can create releases and we can see if this has helped. pipdeptree should be much happier at least.
Let's keep this issue open for the moment.

[mauritsvanrees]

I have created a pre alpha release of Plone 6, which includes the above fixes. See community announcement.
When I pip install this, and try pipdeptree, I get:

$ bin/pipdeptree
Warning!! Cyclic dependencies found:
* plone.app.portlets => plone.app.event => plone.app.portlets
* plone.app.event => plone.app.portlets => plone.app.event
------------------------------------------------------------------------

After that, pipdeptree starts spinning, heating up my processor. I am not sure yet if it will finish.

Anyway: much better. :-)

[ericof]

@mauritsvanrees New PR coming in :-)

[ericof]

plone.app.portlets

• plone.app.event (Remove cyclic dependency with plone.app.event. plone.app.portlets#156)

[mauritsvanrees]

I did bin/pipdeptree --graph-output jpeg > out.jpeg. This took 7 minutes. Result is a 25 MB file, that I cannot even upload here.

I pip installed Zope instead, with pip 21.2.4, and did the same. This took 2 seconds and resulted in 2 MB. See the file here:

[mauritsvanrees]

Eric Steele has released plone.app.portlets==5.0.0a5 with the latest fix. I have updated the constraints and versions on dist.plone.org (but this may be cached for up to a day). I installed this version with pip and ran pipdeptree and it no longer complained about circular dependencies. Yay!

The pipdeptree command keeps spinning though. When I quit, it is busy generating the text output, which will be a lot. If you ask it to output a flat json, this only takes a few second:

bin/pipdeptree --json> flat.json

When you ask it to generate text (no options) or a json tree (--json-tree) it takes minutes, assuming it finishes. When I quit either of those commands, the exit traceback shows it is in a list comprehension in one of the two aux functions in pipdeptree.py. Maybe that can be improved.

I can imagine pip with its new resolver has similar code that could use a speedup.
In the current simple case, where we pip install Plone or Products.CMFPlone, no resolving is is really needed, because everything is presumably pinned. But when you install an add-on on top of this, pip may need to do some resolving.

Anyway, we can probably close this issue, as the circular dependencies are gone. Maybe open up a new issue for speeding up Plone with pip. But that can also be a new issue in the pip repo.