Allows working copy of the Portal #1823

wesleybl · 2024-10-08T12:34:16Z

This is part of: plone/volto#5284

Allows working copy services to be accessed in the Portal. Also returns working copy data in the Portal serialization.

In Plone 6.0, this PR should be tested together with the following PRs:

plone/plone.app.iterate#128
plone/plone.app.versioningbehavior#87
plone/Products.CMFEditions#113

In Plone 6.1, the plone.app.iterate PR should be replaced with:

plone/plone.app.iterate#130

📚 Documentation preview 📚: https://plonerestapi--1823.org.readthedocs.build/

Allows working copy services to be accessed in the Portal. Also returns working copy data in the Portal serialization.

mister-roboto · 2024-10-08T12:34:26Z

@wesleybl thanks for creating this Pull Request and helping to improve Plone!

TL;DR: Finish pushing changes, pass all other checks, then paste a comment:

@jenkins-plone-org please run jobs

To ensure that these changes do not break other parts of Plone, the Plone test suite matrix needs to pass, but it takes 30-60 min. Other CI checks are usually much faster and the Plone Jenkins resources are limited, so when done pushing changes and all other checks pass either start all Jenkins PR jobs yourself, or simply add the comment above in this PR to start all the jobs automatically.

Happy hacking!

wesleybl · 2024-10-08T13:14:31Z

I tried to do a checkout test of the Portal in this class:

plone.restapi/src/plone/restapi/tests/test_services_workingcopy.py

Line 12 in f209cf5

class TestWorkingCopyEndpoint(unittest.TestCase):

The test is as follows:

    def test_workingcopy_checkout_checkin_portal(self):
        # We create the working copy
        response = self.api_session.post(
            "/@workingcopy",
        )

        self.assertEqual(response.status_code, 201)

But I am getting error 500. I went to debug and the error is not in the restapi code. But in Zope. The error occurs when trying to execute this commit:

https://github.com/zopefoundation/Zope/blob/5.10/src/ZPublisher/WSGIPublisher.py#L187

The error is a POSKeyError. I executed the commit outside the try and the traceback was:

*** transaction.interfaces.TransactionFailedError: An operation previously failed, with traceback:

  File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/.buildout/eggs/cp312/waitress-2.1.2-py3.12.egg/waitress/task.py", line 84, in handler_thread
    task.service()
  File "/home/user/.buildout/eggs/cp312/waitress-2.1.2-py3.12.egg/waitress/channel.py", line 428, in service
    task.service()
  File "/home/user/.buildout/eggs/cp312/waitress-2.1.2-py3.12.egg/waitress/task.py", line 168, in service
    self.execute()
  File "/home/user/.buildout/eggs/cp312/waitress-2.1.2-py3.12.egg/waitress/task.py", line 434, in execute
    app_iter = self.channel.server.application(environ, start_response)
  File "/home/user/.buildout/eggs/cp312/WebTest-3.0.0-py3.12.egg/webtest/http.py", line 82, in wrapper
    return self.test_app(environ, start_response)
  File "/home/user/.buildout/eggs/cp312/Zope-5.10-py3.12.egg/ZPublisher/httpexceptions.py", line 30, in __call__
    return self.application(environ, start_response)
  File "/home/user/.buildout/eggs/cp312/Zope-5.10-py3.12.egg/ZPublisher/WSGIPublisher.py", line 390, in publish_module
    with transaction_pubevents(request, response):
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/home/user/.buildout/eggs/cp312/Zope-5.10-py3.12.egg/ZPublisher/WSGIPublisher.py", line 187, in transaction_pubevents
    tm.commit()
  File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_manager.py", line 254, in commit
    return self.manager.commit()
  File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_manager.py", line 133, in commit
    return self.get().commit()
  File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_transaction.py", line 280, in commit
    t, v, tb = self._saveAndGetCommitishError()
  File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_transaction.py", line 273, in commit
    self._commitResources()
  File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_transaction.py", line 456, in _commitResources
    raise v.with_traceback(tb)
  File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_transaction.py", line 430, in _commitResources
    rm.commit(self)
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/Connection.py", line 492, in commit
    self._commit_savepoint(transaction)
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/Connection.py", line 1033, in _commit_savepoint
    obj._p_estimated_size = len(data)
    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/git/buildout.coredev/src/Products.CMFPlone/Products/CMFPlone/Portal.py", line 68, in __setattr__
    if self._tree is not None and name in self:
       ^^^^^^^^^^
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/Connection.py", line 787, in setstate
    p, serial = self._storage.load(oid)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/mvccadapter.py", line 164, in load
    r = self._storage.loadBefore(oid, self._start)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/DemoStorage.py", line 229, in loadBefore
    return self.base.loadBefore(oid, tid)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/DemoStorage.py", line 229, in loadBefore
    return self.base.loadBefore(oid, tid)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/DemoStorage.py", line 229, in loadBefore
    return self.base.loadBefore(oid, tid)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/utils.py", line 281, in __call__
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/MappingStorage.py", line 171, in loadBefore
    raise ZODB.POSException.POSKeyError(oid)
ZODB.POSException.POSKeyError: 0x2a33e463be38bb08

When Plone is running normally this error does not occur, only in testing. Could there be some missing test configuration?

@mauritsvanrees @davisagli @d-maurer @dataflake

d-maurer · 2024-10-08T14:11:00Z

Wesley Barroso Lopes wrote at 2024-10-8 06:14 -0700:

... File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/MappingStorage.py", line 171, in loadBefore raise ZODB.POSException.POSKeyError(oid) ZODB.POSException.POSKeyError: 0x2a33e463be38bb08

You try to load an object state which is not there. Your use of a `DemoStorage` (i.e. a RAM based storage) suggests a test setup with few objects. The oid `0x2a33e463be38bb08` is huge for such a setup. I cannot tell you more. I likely would try to find out where this oid comes from.

wesleybl · 2024-10-08T15:12:32Z

The oid 0x2a33e463be38bb08 is huge for such a setup.

@d-maurer the functionality I'm implementing clones the portal with pickle. See:

https://github.com/plone/Products.CMFEditions/blob/c80bc31af46bff45fee2908878b1f01190fda8d8/Products/CMFEditions/ArchivistTool.py#L210-L229

Usually the Portal is a large object. Could it be that the test configuration is not supporting the clone size?

In real life, would I have to optimize the portal clone, to clone only the attributes that are really needed?

d-maurer · 2024-10-08T16:09:36Z

Wesley Barroso Lopes wrote at 2024-10-8 08:12 -0700:

> The oid `0x2a33e463be38bb08` is huge for such a setup. @d-maurer the functionality I'm implementing clones the portal with pickle. See: https://github.com/plone/Products.CMFEditions/blob/c80bc31af46bff45fee2908878b1f01190fda8d8/Products/CMFEditions/ArchivistTool.py#L210-L229 Usually the Portal is a large object.

Your `oid` is huge; it indicates a setup much larger than a typical Plone site.

Could it be that the test configuration is not supporting the clone size?

I would not expect a `POSKeyError` indicating a RAM (storage) shortage.

In real life, would I have to optimize the portal clone, to clone only the attributes that are really needed?

I am not sure that cloning a portal will work out of the box. At least, the cloned catalog will contain entries refering to the source (and not only entries refering to the clone). They might be other problems. Keep fingers crossed.

wesleybl · 2024-10-08T19:16:28Z

@d-maurer there is already a treatment to prevent child objects of the Portal from being placed in the clone:

https://github.com/plone/Products.CMFEditions/blob/13fe34186f7294113f5c14bac6e8f78170868569/Products/CMFEditions/StandardModifiers.py#L322-L347

The child objects are replaced by an object of the VersionAwareReference class. So my theory that the clone was getting big because of the child objects is wrong.

Checking the traceback, I noticed that the error occurs in the Portal class, in this line:

https://github.com/plone/Products.CMFPlone/blob/6.0.13/Products/CMFPlone/Portal.py#L68

So I printed self before that line. The result was:

...
<PloneSite at plone>
<PloneSite at plone>
<Products.CMFPlone.Portal.PloneSite object at 0x7f697d4581a0 oid 0x1d86b25a16bc20b5 in <ZODB.Connection.Connection object at 0x7f697d4ddb80>>

In other words, the object that caused the error was this:

<Products.CMFPlone.Portal.PloneSite object at 0x7f697d4581a0 oid 0x1d86b25a16bc20b5 in <ZODB.Connection.Connection object at 0x7f697d4ddb80>>

When we had the object:

<PloneSite at plone>

the error did not occur. Does this give any clue as to what might be causing the error?

d-maurer · 2024-10-08T20:17:45Z

Wesley Barroso Lopes wrote at 2024-10-8 12:16 -0700:

... <Products.CMFPlone.Portal.PloneSite object at 0x7f697d4581a0 oid 0x1d86b25a16bc20b5 in <ZODB.Connection.Connection object at 0x7f697d4ddb80>>

OIDs (= "Object IDs") are assigned sequentially; this means, an OID roughly gives a hint on how many objects are in the ZODB (at least if objects have not been massively deleted). The OID "0x1d86b25a16bc20b5" corresponds to "2.127.583.973.977.366.709" or roughly 2 * 10^18. I cannot believe that you really have a ZODB with so many objects. I fear something is very wrong with your setup (and the OIDs are wrong which would correspond to `POSKeyError`s).

... When we had the object: <PloneSite at plone>

This is the `repr` format for an acquisition wrapper; this means, here you do not yet access the base persistent object but the acquisition wrapper wrapping the object at location `plone`. The `<Products.CMFPlone.Portal.PloneSite object at 0x7f697d4581a0 oid 0x1d86b25a16bc20b5 in <ZODB.Connection.Connection object at 0x7f697d4ddb80>` follows the `repr` format for a persistent object (when derived classes do not define `repr` differently). This means that you access here the base portal object (no longer acquisition wrapped). If `pobj` is a persistent object (or an acquisition wrapper thereof), you get its OID with `pobj._p_oid` and its ZODB connection with `poid._p_jar`. You can check whether a ZODB connection `conn` knows about an OID `oid` with `conn[oid]`. This will give you the (unwrapped) object with `oid` or a `POSKeyError` (if the object is not known).

Plone 5.2

d-maurer · 2024-10-09T06:15:17Z

Your use of a DemoStorage (i.e. a RAM based storage) suggests a test setup with few objects. The oid 0x2a33e463be38bb08 is huge for such a setup.

I have been partially wrong: DemoStorage is not a (pure) RAM based storage. Instead it allows to put a RAM based storage (more precisely a MappingStorage) on top of any storage (in particular a non RAM based storage).

Looking more closely at your traceback:
The recursive call DemoStorage.py", line 229, in loadBefore suggests that you have several DemoStorages layered on top of one another. The final traceback line from MappingStorage suggests that you have a MappingStorage at the base of your DemoStorage stack. This would mean that your storage is indeed fully RAM based.

Checking the code, I see that DemoStorage chooses its initial oid randomly: self._next_oid = random.randint(1, 1 << 62). This explains the huge oid value. Using randomness at this point may lead to non-deterministic failures, however, e.g. that the same oid is used for different objects. I suggest you replace line 135 of ZODB.DemoStorage (self._next_oid = random.randint(1, 1 << 62)) by self._next_oid = self.base._next_oid + 0x100000 and check whether this lets your test succeed.

d-maurer · 2024-10-09T06:48:07Z

I have created zopefoundation/ZODB#401

wesleybl · 2024-10-09T12:57:00Z

I suggest you replace line 135 of ZODB.DemoStorage (self._next_oid = random.randint(1, 1 << 62)) by self._next_oid = self.base._next_oid + 0x100000 and check whether this lets your test succeed.

After changing the line, I get the error:

AttributeError: 'MappingStorage' object has no attribute '_next_oid'. Did you mean: 'new_oid'?

I did a dir on self.base:

['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__implemented__', '__init__', '__init_subclass__', '__le__', '__len__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__providedBy__', '__provides__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_commit_lock', '_data', '_last_pack', '_lock', '_ltid', '_oid', '_opened', '_transaction', '_transactions', 'checkCurrentSerialInTransaction', 'cleanup', 'close', 'getName', 'getSize', 'getTid', 'history', 'isReadOnly', 'iterator', 'lastTransaction', 'load', 'loadBefore', 'loadSerial', 'new_oid', 'not_in_transaction', 'opened', 'pack', 'registerDB', 'sortKey', 'store', 'tpc_abort', 'tpc_begin', 'tpc_finish', 'tpc_transaction', 'tpc_vote']

d-maurer · 2024-10-09T13:40:56Z

Wesley Barroso Lopes wrote at 2024-10-9 05:57 -0700:

> I suggest you replace line 135 of ZODB.DemoStorage (self._next_oid = random.randint(1, 1 << 62)) by self._next_oid = self.base._next_oid + 0x100000 and check whether this lets your test succeed. After changing the line, I get the error: ``` AttributeError: 'MappingStorage' object has no attribute '_next_oid'. Did you mean: 'new_oid'? ```

My fault! I had not checked the base implementation. Looking at `DemoStorage.new_oid`: the `_next_oid` is just used as a potential OID candidate; `new_oid` checks that it is currently not used; if it is, then a new candidate is randomly chosen. Thus, at least if the base storage is not independently changed (unlikely in your situation), `DemoStorage`'s OID generation should be safe. Almost surely, my assumption was wrong that the huge OID is related to the problem you have observed.

wesleybl · 2024-10-09T15:17:45Z

@d-maurer I debugged a bit and found out that this error occurs when we have a savepoint and a Plone Site object is involved. It occurs when the code enters here:

https://github.com/zopefoundation/ZODB/blob/5.8.1/src/ZODB/Connection.py#L486-L494

However, the error only occurs in the test and when there is a portal object involved. Even in the test, this code works, if there is no modified Plone Site object.

In the normal execution of an instance, the code does not enter the if, because self._savepoint_storage is None. self._savepoint_storage is not filled, because we have no resources here:

https://github.com/zopefoundation/transaction/blob/4.0/src/transaction/_transaction.py#L618

So the savepoint is not actually created. But in the test we have resources and the savepoint code is executed.

In fact, if I return after this line, the test works.

Do we have a limitation of DemoStorage running savepoint code when we have a Plone Site involved?

d-maurer · 2024-10-09T17:04:14Z

File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/Connection.py", line 1033, in _commit_savepoint
    obj._p_estimated_size = len(data)
    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/git/buildout.coredev/src/Products.CMFPlone/Products/CMFPlone/Portal.py", line 68, in __setattr__
    if self._tree is not None and name in self:
       ^^^^^^^^^^

New hypothesis: The code cited above tries to update the object's size estimation (obj._p_estimated_size = ...). This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But, apparently obj._tree is accessed and results in a ZODB load which apparently fails.

I suggest you try the following:
in Products.CMFPlone.Portal.__setattr__ you replace

        if self._tree is not None and name in self:

by

      if not name.startswith("_p_") and self._tree is not None and name in self:

Apparently the special __setattr__ is there to allow (content) items to be stored in a tree structure (providing support for a huge number of child items). Because the id of content items must not start with _, the startswith("_p_") can be replaced by startswith("_").

wesleybl · 2024-10-09T19:57:08Z

I suggest you try the following:
in Products.CMFPlone.Portal.__setattr__ you replace

   if self._tree is not None and name in self:

by

 if not name.startswith("_p_") and self._tree is not None and name in self:

@d-maurer This worked! Thanks!

I'll try to create a PR to fix this.

Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment)

…ves (#4026) Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment) Co-authored-by: Rohan Shaw <86848116+rohnsha0@users.noreply.github.com>

Branch: refs/heads/master Date: 2024-10-15T22:21:08-07:00 Author: Wesley Barroso Lopes (wesleybl) <wesleybl@gmail.com> Commit: plone/Products.CMFPlone@9a1a62b Avoid POSKeyError when commit occurs and we have savepoint that involves (#4026) Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment) Co-authored-by: Rohan Shaw <86848116+rohnsha0@users.noreply.github.com> Files changed: A news/4026.bugfix M Products/CMFPlone/Portal.py

Allows working copy of the Portal

6b3b52e

Allows working copy services to be accessed in the Portal. Also returns working copy data in the Portal serialization.

Add Changes

7c2ba19

wesleybl requested review from mauritsvanrees, davisagli and djay October 8, 2024 12:40

wesleybl force-pushed the working_copy_portal branch from 7e2f686 to 1ab8ffa Compare October 8, 2024 22:20

Does not return working copy information when serializing Portal in

1ab8ffa

Plone 5.2

wesleybl mentioned this pull request Oct 9, 2024

Avoid POSKeyError when commit occurs and we have savepoint that involves Plone Site plone/Products.CMFPlone#4026

Merged

d-maurer mentioned this pull request Oct 10, 2024

POSKeyError in Connection._commit_savepoint zopefoundation/ZODB#402

Open

This was referenced Oct 10, 2024

Shows working copy options for Plone Site plone/volto#6393

Open

Working copy for Plone Site plone/volto#5284

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allows working copy of the Portal #1823

Allows working copy of the Portal #1823

wesleybl commented Oct 8, 2024 •

edited

Loading

mister-roboto commented Oct 8, 2024

wesleybl commented Oct 8, 2024

d-maurer commented Oct 8, 2024 via email

wesleybl commented Oct 8, 2024

d-maurer commented Oct 8, 2024 via email

wesleybl commented Oct 8, 2024 •

edited

Loading

d-maurer commented Oct 8, 2024 via email

d-maurer commented Oct 9, 2024

d-maurer commented Oct 9, 2024

wesleybl commented Oct 9, 2024

d-maurer commented Oct 9, 2024 via email

wesleybl commented Oct 9, 2024

d-maurer commented Oct 9, 2024

wesleybl commented Oct 9, 2024

Allows working copy of the Portal #1823

Are you sure you want to change the base?

Allows working copy of the Portal #1823

Conversation

wesleybl commented Oct 8, 2024 • edited Loading

mister-roboto commented Oct 8, 2024

wesleybl commented Oct 8, 2024

d-maurer commented Oct 8, 2024 via email

wesleybl commented Oct 8, 2024

d-maurer commented Oct 8, 2024 via email

wesleybl commented Oct 8, 2024 • edited Loading

d-maurer commented Oct 8, 2024 via email

d-maurer commented Oct 9, 2024

d-maurer commented Oct 9, 2024

wesleybl commented Oct 9, 2024

d-maurer commented Oct 9, 2024 via email

wesleybl commented Oct 9, 2024

d-maurer commented Oct 9, 2024

wesleybl commented Oct 9, 2024

wesleybl commented Oct 8, 2024 •

edited

Loading

wesleybl commented Oct 8, 2024 •

edited

Loading