PLIP: Using Dexterity for Plone Site

[jaroel]

PLIP (Plone Improvement Proposal)

Responsible Persons

Proposer: Roel Bruggink

Seconder: Alessandro Pisa

Abstract

Replace the CMFPlone Portal class with a Dexterity implementation.

Motivation

Moving from a CMF item to a Dexterity container will allow us to edit the site root directly using the DX framework including support for behaviors.

This makes it possible to add the rich text behavior to the site root. This way we don’t need to use extra documents to select as default views. This is the last step we need to be able to go with a full folderish objects site and get rid of the “select item as default view” which is a huge usability improvement

Assumptions

Proposal & Implementation

Replace the base class for Products.CMFPlone.Portal.PloneSite with a DX container.
Development is initially done on the latest stable (5.1) branch of Products.CMFPlone. We'll move to unstable (5.2) when this is done or if 5.2 is released, whichever is earlier.

Deliverables

PLIP buildout config can be found here: https://github.com/plone/buildout.coredev/blob/6.0/plips/plip-2454-dx-siteroot.cfg

Jenkins job: https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.8/

Change Products.CMFPlone.Portal.PloneSite from CMF to DX. #2612
Various supportive changes in -at least- plone/plone.app.contenttypes#475 and plone/plone.dexterity#85.

Risks

Currently unknown.

Participants

Roel Bruggink
Alessandro Pisa

[jensens]

How do you plan to migrate the Site Root?

[jaroel]

I haven't given it too much thought atm.
I copied over an existing Plone 5.1 Data.fs into my development stuff and at least it loaded the state correctly from the ZODB, had the correct mro for the Plone Site root and rendered the site just fine.

Something like this should do using a view/script on the zope root for each Plone site:

class MigrateRootToDx(BrowserView):

    def __call__(self):
        from Acquisition import aq_base
        from plone.protect.auto import safeWrite
        request = self.request

        print("Start migration Plone Sites root to DX")
        for site in self.context.objectValues('Plone Site'):
            safeWrite(site, request)
            print("Running for {}".format(site))

            # Initialise the btree.
            if '_tree' not in site.__dict__:
                print("Initialising btrees")
                site._initBTrees()

            # Migrate existing portal content.
            print("Migrate content")
            for obj_meta in site._objects:
                obj_id = obj_meta['id']
                print("Migrating object {}".format(obj_id))
                # Load the content item, tool, or whatever
                obj = getattr(site, obj_id)
                # insert it into the btree.
                # Use _setOb so we don't reindex stuff. the paths stay the same.
                site._setOb(obj_id, aq_base(obj))
                # Drop the attribute
                del site.__dict__[obj_id]

            # Drop no-longer needed bookkeeping
            print("Clean up bookkeeping data")
            del site.__dict__['_objects']
            site._p_changed = True

I do need to check what bookkeeping can be dropped.

edit: I've just tested the above and that seems to Just Works (tm).

>>> sorted(app.Plone.keys())
['_Access_arbitrary_user_session_data_Permission', '_Access_contents_information_Permission', '_Access_inactive_portal_content_Permission', '_Access_session_data_Permission', '_Add_portal_content_Permission', '_Add_portal_folders_Permission', '_Add_portal_member_Permission', '_Allow_sendto_Permission', '_CMFEditions__Access_previous_versions_Permission', '_CMFEditions__Apply_version_control_Permission', '_CMFEditions__Checkout_to_location_Permission', '_CMFEditions__Revert_to_previous_versions_Permission', '_CMFEditions__Save_new_version_Permission', '_Change_local_roles_Permission', '_Content_rules__Manage_rules_Permission', '_Copy_or_Move_Permission', '_Delete_comments_Permission', '_Delete_objects_Permission', '_Delete_own_comments_Permission', '_Edit_comments_Permission', '_FTP_access_Permission', '_List_folder_contents_Permission', '_List_portal_members_Permission', '_List_undoable_changes_Permission', '_Mail_forgotten_password_Permission', '_Manage_properties_Permission', '_Modify_portal_content_Permission', '_Modify_view_template_Permission', '_Portlets__Manage_own_portlets_Permission', '_Portlets__Manage_portlets_Permission', '_Portlets__View_dashboard_Permission', '_Reply_to_item_Permission', '_Request_review_Permission', '_Review_comments_Permission', '_Review_portal_content_Permission', '_Search_ZCatalog_Permission', '_Set_own_password_Permission', '_Set_own_properties_Permission', '_Show_Toolbar_Permission', '_Undo_changes_Permission', '_Use_Database_Methods_Permission', '_Use_external_editor_Permission', '_Use_mailhost_services_Permission', '_Use_version_control_Permission', '_View_Groups_Permission', '_View_History_Permission', '_View_Permission', '_View_management_screens_Permission', '_WebDAV_Lock_items_Permission', '_WebDAV_Unlock_items_Permission', '_WebDAV_access_Permission', '__ZCacheManager_ids__', '__ac_local_roles__', '__ac_roles__', '__allow_groups__', '__annotations__', '__before_publishing_traverse__', '__before_traverse__', '__error_log__', '_components', '_count', '_mt_index', '_owner', '_plone_app_contenttypes__Add_Collection_Permission', '_plone_app_contenttypes__Add_Document_Permission', '_plone_app_contenttypes__Add_Event_Permission', '_plone_app_contenttypes__Add_File_Permission', '_plone_app_contenttypes__Add_Folder_Permission', '_plone_app_contenttypes__Add_Image_Permission', '_plone_app_contenttypes__Add_Link_Permission', '_plone_app_contenttypes__Add_News_Item_Permission', '_plone_app_event__Import_Ical_Permission', '_plone_app_multilingual__Manage_Translations_Permission', '_plone_resource__Export_ZIP_file_Permission', '_plone_resourceeditor__Manage_Sources_Permission', '_properties', '_tree', '_v__providedBy__', 'cmf_uid', 'contributors', 'creation_date', 'creators', 'default_page', 'description', 'effective_date', 'expiration_date', 'format', 'id', 'language', 'modification_date', 'rights', 'subject', 'title']

>>> app.Plone.portal_catalog
<CatalogTool at /Plone/portal_catalog>

>>> app.Plone.portal_catalog()
[<Products.ZCatalog.Catalog.mybrains object at 0x10b27e598>, <Products.ZCatalog.Catalog.mybrains object at 0x10b27e0b8>, <Products.ZCatalog.Catalog.mybrains object at 0x10b258668>, <Products.ZCatalog.Catalog.mybrains object at 0x10b258940>, <Products.ZCatalog.Catalog.mybrains object at 0x10b258bb0>]

>>> app.Plone.portal_catalog()[0]
<Products.ZCatalog.Catalog.mybrains object at 0x10b258328>

>>> app.Plone.portal_catalog()[0].getObject()
<Document at /Plone/front-page>

>>> 'portal_catalog' in app.Plone.__dict__
False

[jaroel]

We'll need to add the Plone Site FTI and fix the edit method alias when upgrading.
Need to check plone.app.multilingual as we won't have a front-page object anymore.

[ebrehault]

@jaroel The FWT is very positive about this PLIP.

It would be useful to get more information about the motivations (what could be the benefits? What kind of behaviors could be usefully used on the Plone Site object) as it would help to communicate about this important improvement.

Note: we will vote on the PLIP during our next meeting.

[sunew]

for one, selectablecontrainstypes behavior would be super useful:)

[robgietema]

This makes it possible to add the rich text behavior to the site root. This way we don’t need to use extra documents to select as default views. This is the last step we need to be able to go with a full folderish objects site and get rid of the “select item as default view” which is a huge usability improvement. I can second this proposal.

[ebrehault]

@jaroel, the @plone/framework-team approved this PLIP

[jaroel]

@ebrehault that's great news, thanks!

[tkimnguyen]

How does this work? "This is the last step we need to be able to go with a full folderish objects site and get rid of the 'select item as default view'" In CastleCMS we got rid of default items by using Mosaic and including a rich text tile on the Folder content type default layout

[tisto]

@jaroel would you mind giving us an update on the current status? What is missing before we can review your code? Do you plan to integrate this in Plone 5.2 or later?

[tisto]

@tkimnguyen in Plone-React we got rid of default items as well. The new tiles-based "Page" type is folderish by default as well.

[pbauer]

@tkimnguyen see https://github.com/plone/plone-react/blob/master/api/buildout.cfg#L24

[jaroel]

@tisto In practice it seems to work just fine. I'm resolving the test failures when I have time.
See https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot/ for the current state.

You can check the changes done so far using the PLIP buildout config https://github.com/plone/buildout.coredev/blob/5.1/plips/plip-2454-dx-siteroot.cfg .

I wouldn't see this as a blocker for 5.2. I'd not even consider this for 5.2, unless people have used this in production somewhere.

[jaroel]

@gforcada or @tisto I'm getting timeout accessing the ftp server on jenkins (https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot/12/testReport/plone.app/testing/layers_rst/)
Any idea what could be going on there?

[jaroel]

FYI I'm back on this. Primary goal is to make this work for 5.2+, as that seems stable enough now.

[tisto]

@jaroel we do not plan to do a 5.3 release so this would be Plone 6 then. I added the label...

[jaroel]

@jensens FYI:

1. So far bin/instance starts and I have a functional Plone site root.
2. The tests failures from plone/plone.supermodel@c1822bb are back
3. https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.8/ has some failures
4. The ZMI (/manage) doesn't reflect the normal interface for the site root
   5. The Products.CMFEditions failures make me sad. They have disappeared somehow, which makes me sad

[esteele]

What's the current status of this? It looks like tests are currently green.

[jaroel]

I've just updated the branches in the repos I needed.
The "upgrade script" #2454 (comment) should be added somewhere. This could be in Products.CMFPlone.Portal.__getstate__ or as a normal upgrade step.
The FWT should review what has been done, and people must test it with actual production data.

To be honest I don't really have time nor incentive currently to work on this except for once in a blue moon. I'm fairly certain it would be more effective if someone like @ale-rt would champion this PLIP.

[ale-rt]

I will work on this at the Not-an-Alpine-City-Sprint

[pbauer]

I already used Pone 6 master for that test but I'll test another project which still runs on Plone 5.2

[pbauer]

I just tried the upgrade using @@plone-upgrade with a project that ran 5.2.4 before and it worked like a charm. I updated the plip-branches locally to the current master of Plone 6. The project has no plone.app.multilingual.

[tisto]

@ale-rt we did not discuss this PLIP at our last FWT meeting. @mauritsvanrees would like to start doing a first (internal) release of Plone 6. Can you give us a heads up about the status here,? If needed, we could schedule a @plone/framework-team meeting on short notice to push this further in order to get this into the first release.

[mauritsvanrees]

I upgraded a Plone 6 site to dx site root and it seemed fine.
After the upgrade, the front-page content item is still there and is used as default page. I wondered if its text would be taken over, and maybe even the front-page itself removed, but this is not the case. Current state is probably best.
Creating a new site worked fine as well.

Two minor things:

1. Currently the homepage shows the text field plus the contents of the root (news, events, Members). If you only want to show the text field and no contents, there is currently no way.
2. The plone.excludefromnavigation behavior is enabled, but when you check this field in the edit form, it has no effect. We should probably just remove this behavior.

For the first one, we could make a copy of document_view and make this available in the view methods. Or is there some basic view which I am not aware of that renders all defined fields?
Probably a simple siteroot_view.pt would be fine. That also gives a nice way for integrators to override this with z3c.jbot.
For me, it is fine to do this after things get merged.

The excludefromnavigation behavior is probably best removed now, unless someone sees a need for this anyway.

[mauritsvanrees]

We could add the versioning behavior, though I don't see this having an effect when I try it, so this might need changes elsewhere.

We could add the constraintypes behavior. This seems to work well when I try it.

Both could be done later, or we leave this as an option for the user.

[mauritsvanrees]

I created two PRs that were missing. Related PRs are now:

• PLIP 2454 DX Plone Site #2612
• Dx container site root plone.app.contenttypes#475
• Dx siteroot plone.app.upgrade#256

Plus the following PRs which can maybe be merged separately because they seem fine for current Plone as well:

• Updated tests to not fail when the Plone Site root is dexterity. plone.restapi#1202
• Dx container site root plone.app.multilingual#336
• Dx siteroot plone.app.testing#60
• Dx container site root plone.dexterity#85
• Dx siteroot plone.supermodel#27

If we merge the PRs which can be done separately, it makes the rest of the PLIP smaller.
[Update: most of these have been merged.]

Note that some branches were created in 2019, and master has been merged into them several times, which makes for an ugly and confusing history. So it could be an option to create fresh branches, or do a rebase. I did a fresh branch for plone.restapi, as the only change left was in two lines.

[mauritsvanrees]

I have updated the PLIP config to only test the three remaining packages with branches.
I have started the three PLIP jobs on Jenkins, on 3.7, 3.8 and 3.9.
I have deleted the 3.6 PLIP project, because Plone 6 no longer supports it.

[mauritsvanrees]

The three jobs all have five the same failures in the robot tests. So this may need some work. Locally the failing ones pass though.
The 3.7 job even has 34 robot failures, I have restarted this one.

See the 3.9 job.
Four are in the user schema editor, where for a checkbox we get:

is not clickable at point (1771, 2208).
Other element would receive the click: <div class="plone-toolbar-container">

See the screenshot, which looks very wrong: the toolbar is visible inside a popup/overlay. When I try it locally, it looks fine though.

The fifth is for the scenario "A page is opened to edit in TinyMCE" where we get:

Button with locator 'css=.pattern-relateditems-container button.favorites' not found.

See screenshot

[mauritsvanrees]

I have restarted the 3.7 job and it again had the same 34 robot failures.
I have restarted the 3.8 job and it again had the same 5 robot failures.
Jenkins did not do much at the time.
So there are no random failures, but real ones.

[mauritsvanrees]

Sorry, I lost sight of the plone.supermodel PR. I added it to the list above now. Can be merged separately. @ale-rt has re-added it to the plip config file and this should fix the PLIP build.

[ale-rt]

Yes, all the branches are green right now:

• https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.7/
• https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.8/
• https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.9/

[ale-rt]

@ale-rt we did not discuss this PLIP at our last FWT meeting. @mauritsvanrees would like to start doing a first (internal) release of Plone 6. Can you give us a heads up about the status here?

@tisto this PLIP gets periodically green.

For me it is even good to be merged now, but nobody did a formal review of the 4 branches:

• PLIP 2454 DX Plone Site #2612
• Dx siteroot plone.app.upgrade#256
• Dx siteroot plone.supermodel#27
• Dx container site root plone.app.contenttypes#475

I never used this code on production, but using this plip branch I could easily migrate and run simple Plone 5.2 websites.
If you ask me I would merge all the PRs right now (this will target the first alpha version...).
I would also like to hear other people opinions.

If needed, we could schedule a @plone/framework-team meeting on short notice to push this further in order to get this into the first release.

Unluckily I think some FWT people are now on vacation, that is why we agreed to meet next time the 7th of September (see https://community.plone.org/t/fwt-meeting-minutes-2021-07-20/14110).
Anyway if other @plone/framework-team people want to organize a meeting on a short notice I will be fine with that :)

[tisto]

@ale-rt let's try to get a formal review of the four branches and feedback from as many @plone/framework-team members as possible. I'd be open to schedule a call on short notice (tomorrow?).

Maybe if all available FWT members and the release managers @mauritsvanrees @esteele agree, we can merge this PLIP and make a holiday exception.

I guess the only way to properly test this is to get it into Plone 6 alpha.

[ale-rt]

Congratulation @jaroel!
As agreed with the @plone/framework-team in the last meeting this PLIP can be merged.
This requires merging:

• PLIP 2454 DX Plone Site #2612
• Dx container site root plone.app.contenttypes#475
• Dx siteroot plone.app.upgrade#256

I currently cannot do that myself because the merge button is disabled for me in #2612.
I think the reason is the failing Python 3.6 build, which should actually be ignored.
CC @mauritsvanrees

[jaroel]

Thank you @ale-rt for your help in landing this PLIP!
Does this mean I need to press 'merge' or what's the next step?

[ale-rt]

I have no power to merge #2612 while a test is still failing, even if it is outdated.
I bet someone has the power to do it :)

[mauritsvanrees]

I have merged all three remaining PRs.

Great job, Roel! Thanks for taking over, Alessandro! Thanks to all others who helped, reviewed, tested!

[jensens]

For information on how to upgrade existing Plone 6 sites. please read https://community.plone.org/t/upgrade-plone-6-sites-dx-site-root/14303