PLIP: Update Zope Dependencies

[thet]

Proposer : Johannes Raggam

Seconder : Philip Bauer

Abstract

This PLIP is about updating Plone's framework dependencies (ZCA, ZTK and Zope) to the latest possible versions.

Motivation

Plone 5 still uses ages old ZTK dependencies and Zope2. While the ZTK dependencies are still actively developed and most of them already Python 3 compatible, the development for Zope 2 has nearly stopped. The successor of Zope2 is Zope 4. The major release version indicates that backwards incompatible refactoring has happened: long deprecated stuff was removed, some performance improvements and bugfixes were done, a few features were added.
To be able to move forward with Plone, we have to modernize our framework dependencies. Also, this PLIP is a precondition for Python 3 compatibility.

Assumptions

• We are not able to switch to Pyramid / SubstanceD soon.
• Modernizing our framework dependencies will help with such a possible switch in the future.
• A recent set of ZTK packages is a precondition for Python 3 compatibility.
• Patching and hacking the ZPublisher is better done with a cleaned up version.

Proposal & Implementation

PLIP configuration: https://github.com/plone/buildout.coredev/blob/5.0/plips/plip-zope4.cfg
Jenkins PLIP job: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/

The PLIP buildout configuration extends versions.cfg and sources.cfg from the Zope project. We added the option auto-checkout = * to check out all sources with mr.developer to get the very latest versions for each dependency. Only some test dependencies are installed as egg. For easier tracking, the eggs are installed in the plips/eggs folder.

We started to fix test failures and created branches and pull requests for all of our changes. Not everything is pull-requested yet. Many of the failures came from moved imports, removed functionality (e.g. no doctest suite any more in zope.testing, as we can use the stdlib for that) but also bugfixes in production code. E.g:

• Compatibility with PythonScripts zopefoundation/zope.publisher#10
• zopefoundation/Zope@3320082
• Interface comparison with emtpy __name__ reports incorrect equality zopefoundation/zope.interface#31

All fixes are made backwards compatible for our current set of dependencies.

At the Alpine City Sprint most of the failures could be fixed. There are only a few test failures left. Also, there are a lot of test isolation problems, but that's another story.

Two packages are not yet updated: Products.ZCatalog and ZODB 4. We still have to decide, if these packages are part of this PLIP. Updating them involves finding a solution for the "ZCatalog Regression: indexing None Values" problem: zopefoundation/Products.ZCatalog#5

We propose this PLIP for inclusion in Plone 6.

Deliverables

• Test fixes.
• Bug fixes.
• Green Jenkins build.
• Upgrade guide based on changes on packages within this PLIP.
• Check monkey patches for their compatibility with Zope 4.

- [ ] Replace dependency on "Zope2" with "Zope". Eventually provide a Zope2 shim. - not necessary - Zope version 4.0 is still named Zope2.

Risks

Zope 4 is currently in alpha state, there is no release yet and development activity is - as with Zope2 - not very high.
We need to show more responsibility for Zope 2 and related packages.

Participants

• Johannes Raggam ( @thet / PLIP proposer / Alpine City Sprint Zope 4 Team)
• Philip Baiuer ( @pbauer / PLIP seconder / Alpine City Sprint Zope 4 Team)
• Gil Forcada ( @gforcada / Jenkins Zope 4 PLIP Job)
• Maurits van Rees ( @mauritsvanrees / Tracking down and fixing Compatibility with PythonScripts zopefoundation/zope.publisher#10 )
• Jens Klein ( @jensens / Tracking down Zope 2 product initialization bug: zopefoundation/Zope@3320082 )
• Alexander Loechel ( @loechel / Began working on Products.RestrictedPython bringing to Python 3. Not part of this PLIP)

Python 3

Python 3 compatibility is not part of the PLIP. A major showstopper for Python 3 compatibility is Products.RestrictedPython. @loechel worked on that during the Alpine City Sprint 2016, which resulted in this document: zopefoundation/RestrictedPython@45c50cb
Also, ZODB 4 is a precondition for Python 3 compatibility.

Notes

Some experiments to install Plone 5 on Zope 4 started back in 2013: https://dev.plone.org/ticket/13822
At the Plone conference 2015 I demoed a running installation of Plone 5 on Zope 4 with the package: https://github.com/collective/experimental.zope4 and started to fix some failing tests.
A major effort on working on this PLIP was done at the Alpine City Strategic Sprint 2016 in Innsbruck: http://www.coactivate.org/projects/alpinecitysprint-2016
Related to this PLIP: "WIP try ZODB3" plone/buildout.coredev#137

[pbauer]

\o/ tests are finally green: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/16/console

There are some pull-request to be made for plone and zope-packages and at least one issue needs to be looked at (plone/plone.supermodel@8a4d202).

The plip-job uses source-checkouts for everything but the following:

BTrees = 4.2.0
Babel = 1.3
Chameleon = 2.24
FormEncode = 1.3.0
Markdown = 2.6.5
Pillow = 3.1.0
Products.PDBDebugMode = 1.3.1
Products.PloneTestCase = 0.9.18
Products.SecureMailHost = 1.1.2
Products.SiteErrorLog = 2.13.2
Products.ZSQLMethods = 2.13.4
PyYAML = 3.11
Pygments = 2.0.2
RestrictedPython = 3.6.0
Unidecode = 0.4.1
WebOb = 1.4
ZConfig = 3.0.4
ZODB = 4.2.0
argh = 0.26.1
argparse = 1.3.0
coverage = 3.7.1
cssmin = 0.2.0
cssselect = 0.9.1
decorator = 4.0.6
docutils = 0.12
feedparser = 5.2.1
five.globalrequest = 1.0
future = 0.15.2
interlude = 1.3.1
lxml = 3.5.0
mailinglogger = 3.8.0
manuel = 1.8.0
mechanize = 0.2.5
mock = 1.0.1
mocker = 1.1.1
pathtools = 0.1.2
persistent = 4.1.1
plone.mocktestcase = 1.0b3
ply = 3.4
python_dateutil = 1.5
python_gettext = 3.0
python_openid = 2.2.5
pytz = 2015.7
robotframework = 2.8.7
robotframework_debuglibrary = 0.3
robotframework_ride = 1.3
robotframework_selenium2library = 1.7.4
robotsuite = 1.7.0
roman = 1.4.0
selenium = 2.46.0
setuptools = 19.6.2
simplejson = 3.8.1
six = 1.9.0
slimit = 0.8.1
sourcecodegen = 0.6.14
transaction = 1.4.4
unittest2 = 0.8.0
watchdog = 0.8.3
wsgi_intercept = 0.4
z3c.caching = 2.0a1
z3c.coverage = 2.0.3
z3c.objpath = 1.1
z3c.pt = 3.0.0a1
z3c.template = 2.0.0
z3c.zcmlhook = 1.0b1
zc.buildout = 2.5.0
zc.lockfile = 1.1.0
zc.recipe.egg = 2.0.3
zc.relation = 1.0
zc.sourcefactory = 0.7.0
zdaemon = 4.1.0
zodbpickle = 0.6.0
zope.app.applicationcontrol = 3.5.10
zope.app.appsetup = 3.14.0
zope.app.authentication = 3.8.0
zope.app.basicskin = 3.5.1
zope.app.broken = 3.6.0
zope.app.component = 3.9.3
zope.app.container = 3.9.2
zope.app.content = 3.5.1
zope.app.debug = 3.4.1
zope.app.dependable = 3.5.1
zope.app.error = 3.5.3
zope.app.exception = 3.6.3
zope.app.file = 3.6.1
zope.app.folder = 3.5.2
zope.app.form = 4.0.2
zope.app.generations = 3.6.1
zope.app.http = 3.9.0
zope.app.i18n = 3.6.4
zope.app.intid = 3.7.1
zope.app.locales = 3.6.2
zope.app.localpermission = 3.7.2
zope.app.pagetemplate = 3.11.2
zope.app.principalannotation = 3.7.0
zope.app.publication = 3.12.0
zope.app.publisher = 3.10.2
zope.app.renderer = 3.5.1
zope.app.rotterdam = 3.5.3
zope.app.schema = 3.5.0
zope.app.security = 3.7.5
zope.app.testing = 3.7.8
zope.app.wsgi = 3.9.3
zope.app.zcmlfiles = 3.7.1
zope.app.zopeappgenerations = 3.5.1
zope.broken = 3.6.0
zope.sequencesort = 4.0.1
zope.testbrowser = 3.11.1
zope.untrustedpython = 4.0.0

[pbauer]

FWIW: I just tested a current Plone5-based project of ours with the versions/checkouts of this PLIP and found no issue. Needs more review though.

[mcdonc]

FWIW, if you're going to refactor ZPublisher a bit, you might take a look at the (very aged, but does it matter?) repoze.zope2, which attempted to do just that:

http://svn.repoze.org/repoze.zope2/trunk/

I started to write Pyramid shortly after this, as I felt that I was never going to be able to factor the publisher sufficiently while maintaining bw compat, so I figured "nuke it from orbit and start over" ;)

[thet]

@pbauer 👍 👍
most of the eggs listed above are only test dependencies.

@mcdonc tnx for the pointer! we will look at it. does repoze.zope2 use WebOb instead of Zope's Request/Response objects? That would be interesting too...

[mcdonc]

No it uses Zope request/response.

[thet]

Unmerged Pull Requests

#1370
zopefoundation/zope.publisher#10
zopefoundation/five.customerize#1

Merged Pull Requests

plone/plone.api#301
plone/plone.app.caching#21
plone/plone.app.contenttypes#317
plone/plone.app.customerize#8
plone/plone.app.dexterity#205
plone/plone.app.i18n#4
plone/plone.app.imaging#25
plone/plone.app.layout#77
plone/plone.app.lockingbehavior#5
plone/plone.app.testing#20
plone/plone.app.upgrade#66
plone/plone.app.users#53
plone/plone.app.z3cform#38
plone/plone.autoform#10
plone/plone.behavior#7
plone/plone.dexterity#50
plone/plone.formwidget.namedfile#17
plone/plone.indexer#3
plone/plone.memoize#3
plone/plone.registry#7
plone/plone.recipe.zope2instance#20
plone/plone.resource#9
plone/plone.rfc822#1
plone/plone.supermodel#13
plone/plone.testing#16
plone/plone.testing#18
plone/Products.Archetypes#51
plone/Products.ATContentTypes#30
plone/Products.PortalTransforms#13
zopefoundation/Zope#45
https://github.com/zopefoundation/Products.ZCatalog/tree/thet-zope4
zopefoundation/five.pt#6

[pbauer]

For now we need to skip the following tests since they fail using Zope4:

• plone.app.lockingbehavior.tests.test_locking.test_lockablebehavior
• plone.dexterity.tests.test_schema.test_portalTypeToSchemaName_looks_up_portal_for_prefix
• plone.dexterity.tests.test_utils.test_getAdditionalSchemata
• Products.CMFPlone.controlpanel.tests.test_controlpanel_browser_maintenance.test_maintenance_pack_database

Also this pull adds some Zope 4 specific case handling, which should be reviewed again:
https://github.com/plone/plone.supermodel/pull/13/files

[ebrehault]

This PLIP is approved for Plone 6, see https://community.plone.org/t/framework-team-meeting-minutes-2016-03-01/1663

[hvelarde]

awesome!

[thet]

Related:

• Most importantly, test isolation problems: Test isolation problems buildout.coredev#179

Also:

• Remove monkey patches from plone.protect 3.X: Remove monkey patches from plone.protect 3.X #1335
• WIP: Zope2: WIP: Zope2 buildout.coredev#165 (Read: forward Port changes from Zope 2.13 to Zope 4)

[pbauer]

I removed the Zope4-reference from the name of the PLIP since the version-bump will only add another fun layer of confusion that we struggled with since the release of the ill-named Zope3.

[thet]

The plip configuration file is still named plip-zope4.cfg. Maybe it's better to keep it as it is, since we would need to update the jenkins PLIP job too.

[pbauer]

Sure. That is way less visible than a PLIP-title.

[hannosch]

I just spend a bit of time on ZCatalog. I merged the commit lingering on the https://github.com/zopefoundation/Products.ZCatalog/tree/thet-zope4 branch.

I also fixed the problem with zopefoundation/Products.ZCatalog#5 - by ignoring None values being indexed, rather than raising TypeErrors.

Both of these changes are in a ZCatalog 3.1.1 release. That release does depend on ZODB 4, so only use it if you want to do the ZODB 4 migration at the same time.

[thet]

Thanks @hannosch ! That clears the way for the important ZODB updates...
Are there incompatibilities with ZODB 4? AFAIK the pickle support in Python 3 has changed, but ZODB still works transparent regarding to the pickle format and still supports Python 2 and 3.

[hannosch]

@thet sorry, got confused with Python 2 to 3 for a moment. IIRC ZODB 4 will need a manual migration when moving from an existing database created under Python 2 to code running under Python 3. But that should be the only one, and it's not relevant here, as Python 3 support is still a ways off.

[gforcada]

@thet @hannosch there is https://pypi.io/project/zodb.py3migrate/ to convert ZODB databases to python 3

[thet]

@hannosch fine! although, hopefully Python 3 will get relevant for Plone soon :)

[pbauer]

What about ZODB 5? Did anyone review the changes that @jimfulton did and what they mean in the context of Plone? See https://github.com/zopefoundation/ZODB/blob/master/CHANGES.rst. Same goes for ZEO 5.

[jimfulton]

Note that ZODB 5 and ZEO 5 are still in alpha. ZODB 5 changes are mainly internal.

ZEO 5 is still evolving, but will provide some significant advantages, including SSL and prefetch.

I think it's a little early to worry about ZODB 5 and ZEO 5.

Also note that RelStorage is being actively developed again.

[pbauer]

The start of a upgrade-guide is here: https://docs.google.com/document/d/1T-E6qD7BoTrebk9L6F1EC0qMvqSOAu0987av6wUZ7nY/edit#

In there we collect everything that need to be documented and move it to their respective places after it looks complete.

[pbauer]

After much work was done during and after the Apline City Sprint (https://www.starzel.de/blog/towards-plone-6) the tests for this PLIP are now green (http://jenkins.plone.org/view/PLIPs/job/plip-zope4/). Remaining tasks will be defined and worked on at PLOG and at the the Zope 2 Resurrection Sprint (https://www.meetup.com/de-DE/Zope-Sprint/events/237672812/).

[pbauer]

Zope 4.0b1 has been released after the great work everyone did at the sprint in Halle last week! 🎉

I would love to get this PLIP into a state where we can be productively work on it during the conference-sprint.

One main issue at the moment is that our branch plonezope4 of Zope (zopefoundation/Zope#174) needs to be separated into pull-requests. @MrTango started working on this at the last sprint but it would be more productive if the authors of the relevant changes @thet, @davisagli, @mauritsvanrees, @MatthewWilkes, @MrTango had a look at their changes and created pull requests if they are still valid. There are only 13 commits of which some are already obsolete and I guess it is doable.

I'll buy nice and cold 🍺 for each of you once these commits are resolved.

Today I did some merging, cleanup and rebased most branches related to this PLIP. I also changed the PLIP-config to no longer checkout all packages but use released packages wherever possible. That saves a lot of time when running the buildout.

And now we can also look at the many new and exciting test-failures in http://jenkins.plone.org/view/PLIPs/job/plip-zope4/ 😉

[pbauer]

I put in some more work and fixed a good couple of things. Among the exciting new issues: Zope now uses new style classes, most tools from CMFCore are now utilities and brains cannot acquire the REQUEST.

Many of the remaining issues are related to plone/Products.PlonePAS#24 (anyone???) and zopefoundation/Products.ZCatalog#27

[pbauer]

I fixed the issues with PlonePAS (ugly but working for now: plone/Products.PlonePAS#25).

Now I see a lot of testbrowser-failures, many of which come from CSRF-protection where we get the confirm-action view instead of the expected URL. One rather simple such test is test_rename_form_cancel from plone.app.content. It passes for AT but fails for DX.

Can anyone give me hint about how to prevent that (except adding _authenticator to all the old tests)?

[mauritsvanrees]

plone.protect has a suggestion, see the paragraph 'Fixing CSRF Protection failures in tests' on PyPI:

from plone.protect.authenticator import createToken
self.request = self.layer['request']
self.request.form['_authenticator'] = createToken()
# and maybe this:
self.request.environ['REQUEST_METHOD'] = 'POST'

That would require a change for each failing test, though hopefully in a shared setUp method this might solve several failures at once.

An alternative might be to set PLONE_CSRF_DISABLED=true in the environment of the OS. You could try that out first: export PLONE_CSRF_DISABLED=true; bin/test -s some.package and see if that helps. If it does, then you can unset the variable again, and temporarily set it in some tests with a contextmanager.

Possibly, such a contextmanager could be added to plone.protect or plone.testing or plone.app.testing so the code is shared.

[pbauer]

@mauritsvanrees I'm aware of these options but to lazy to to that. What I wonder is which change might be responsible for these issues appearing now and if we can work around it there. It did not happen 5 month ago when we had all test green (with source-checouts of zope.testbrowser and plone.testing).

[hannosch]

@pbauer I'd check if the code in https://github.com/plone/plone.transformchain/blob/master/plone/transformchain/zpublisher.py is still working as expected. Publication events are now emitted at a different place in the WSGI publisher. It should be more correct and fixes other issues. But maybe that's caused an issue for the transformchain code or something specific for testbrowser tests.

[pbauer]

All branches in here are merged. Thanks so much to everyone helped working on it!. We have still 5 failing tests left in the new 5.2-branch of the coredev, that we will fix in the next couple of days. What remains to be done is to:

• write upgrade-steps that deal with the changes (e.g. Migrate Plone site from Zope2 2.13 to Zope2 4.0b3.dev0 zopefoundation/Zope#205) and
• write the upgrade-guide for changes related to this PLIP for what is now Plone 5.2 (started in https://docs.google.com/document/d/1T-E6qD7BoTrebk9L6F1EC0qMvqSOAu0987av6wUZ7nY)

[jensens]

I think except documentation this can be seen as done. Documentation has its own ticket #1940
So, I close this!