Skip to content

Commit

Permalink
[fc] Repository: plone.app.users
Browse files Browse the repository at this point in the history
Branch: refs/heads/master
Date: 2022-03-08T20:46:41+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/plone.app.users@e99f375

Fixed tests when run with zope.component 5+.

Sample test failure:

```
File "/Users/maurits/community/plone-coredev/6.0/src/plone.app.users/plone/app/users/tests/password.rst", line 27, in password.rst
Failed example:
    browser.open('http://nohost/plone/' + view_name)
Expected:
    Traceback (most recent call last):
    ...
    zExceptions.unauthorized.Unauthorized: ...You are not authorized to access this resource...
Got:
    Traceback (most recent call last):
      File "/Users/maurits/.pyenv/versions/3.9.9/lib/python3.9/doctest.py", line 1334, in __run
        exec(compile(example.source, filename, "single",
      File "&lt;doctest password.rst[8]&gt;", line 1, in &lt;module&gt;
        browser.open('http://nohost/plone/' + view_name)
      File "/Users/maurits/shared-eggs/cp39/zope.testbrowser-5.5.1-py3.9.egg/zope/testbrowser/browser.py", line 256, in open
        self._processRequest(url, make_request)
      File "/Users/maurits/shared-eggs/cp39/zope.testbrowser-5.5.1-py3.9.egg/zope/testbrowser/browser.py", line 282, in _processRequest
        resp = make_request(reqargs)
      File "/Users/maurits/shared-eggs/cp39/zope.testbrowser-5.5.1-py3.9.egg/zope/testbrowser/browser.py", line 253, in make_request
        return self.testapp.get(url, **args)
      File "/Users/maurits/shared-eggs/cp39/WebTest-3.0.0-py3.9.egg/webtest/app.py", line 324, in get
        return self.do_request(req, status=status,
      File "/Users/maurits/shared-eggs/cp39/zope.testbrowser-5.5.1-py3.9.egg/zope/testbrowser/browser.py", line 93, in do_request
        response = super(TestbrowserApp, self).do_request(req, status,
      File "/Users/maurits/shared-eggs/cp39/WebTest-3.0.0-py3.9.egg/webtest/app.py", line 620, in do_request
        res = req.get_response(app, catch_exc_info=True)
      File "/Users/maurits/shared-eggs/cp39/WebOb-1.8.7-py3.9.egg/webob/request.py", line 1309, in send
        status, headers, app_iter, exc_info = self.call_application(
      File "/Users/maurits/shared-eggs/cp39/WebOb-1.8.7-py3.9.egg/webob/request.py", line 1278, in call_application
        app_iter = application(self.environ, start_response)
      File "/Users/maurits/shared-eggs/cp39/WebTest-3.0.0-py3.9.egg/webtest/lint.py", line 196, in lint_app
        iterator = application(environ, start_response_wrapper)
      File "/Users/maurits/shared-eggs/cp39/plone.testing-8.0.3-py3.9.egg/plone/testing/_z2_testbrowser.py", line 39, in wrapped_func
        return func(*args, **kw)
      File "/Users/maurits/shared-eggs/cp39/plone.testing-8.0.3-py3.9.egg/plone/testing/_z2_testbrowser.py", line 66, in __call__
        wsgi_result = publish(environ, start_response)
      File "/Users/maurits/community/plone-coredev/6.0/src/Zope/src/ZPublisher/WSGIPublisher.py", line 391, in publish_module
        request.close()
      File "/Users/maurits/community/plone-coredev/6.0/src/Zope/src/ZPublisher/BaseRequest.py", line 215, in close
        notify(EndRequestEvent(None, self))
      File "/Users/maurits/shared-eggs/cp39/zope.event-4.5.0-py3.9.egg/zope/event/__init__.py", line 32, in notify
        subscriber(event)
      File "/Users/maurits/shared-eggs/cp39/zope.component-5.0.1-py3.9.egg/zope/component/event.py", line 27, in dispatch
        component_subscribers(event, None)
      File "/Users/maurits/shared-eggs/cp39/zope.component-5.0.1-py3.9.egg/zope/component/_api.py", line 134, in subscribers
        return sitemanager.subscribers(objects, interface)
      File "/Users/maurits/shared-eggs/cp39/zope.interface-5.4.0-py3.9-macosx-10.15-x86_64.egg/zope/interface/registry.py", line 448, in subscribers
        return self.adapters.subscribers(objects, provided)
      File "/Users/maurits/shared-eggs/cp39/zope.interface-5.4.0-py3.9-macosx-10.15-x86_64.egg/zope/interface/adapter.py", line 895, in subscribers
        subscriptions = self.subscriptions([providedBy(o) for o in objects], provided)
      File "/Users/maurits/shared-eggs/cp39/zope.interface-5.4.0-py3.9-macosx-10.15-x86_64.egg/zope/interface/adapter.py", line 877, in _uncached_subscriptions
        if order &gt;= len(byorder):
      File "/Users/maurits/.pyenv/versions/3.9.9/lib/python3.9/collections/__init__.py", line 1169, in __len__
        return len(self.data)
      File "/Users/maurits/shared-eggs/cp39/ZODB-5.6.0-py3.9.egg/ZODB/Connection.py", line 785, in setstate
        raise ConnectionStateError(msg)
    ZODB.POSException.ConnectionStateError: Shouldn't load state for persistent.list.PersistentList 0x3953f62e87808eb2 when the connection is closed
```

Should be somehow caused by this zope.component PR:
zopefoundation/zope.component#51

From what I can follow from the traceback and running around in the pdb, it goes wrong in this part of `ZPublisher/BaseRequest.py`:

```
    def close(self):
        try:
            notify(EndRequestEvent(None, self))
        finally:
            # subscribers might need the zodb, so `clear` must come afterwards
            # (since `self._held=None` might close the connection, see above)
            self.clear()
```

The notify fails, even when I comment out the two subscribers that I found (`Products.Five` and `plone.app.contentrules`).
Probably some other code has already called `self.clear()` before this.
A reason for this can be that several subrequests are made for the Diazo theme, and this seems to call `self.clear()` directly.
Whatever the reason, the comment is true: subscribers might need the zodb.
In fact, with zope.component 5, the list of subscribers itself, even when it is empty, is something that we get from the zodb.
So it fails here.

When I run `bin/instance fg` and visit the same page anonymously (`@@change-password`) it works fine:
I am Unauthorized and get redirected, which is also what the test expects.
So in practice all seems fine.

So what somehow triggers the ZODB error, is that this is a test, and the first browser visit is to a page that raises an Unauthorized.
The fix/workaround is to first visit the root of the site.  Then somehow all is well.

Note: last time I tried this, most tests in most packages passed just fine.
`plone.app.users` was an exception, and there were about two other packages I think.
I will follow up with more PRs, but this one should be fine, also for use with our current outdated zope.component 4.6.2.

Files changed:
A news/500.bugfix
M plone/app/users/tests/password.rst
M plone/app/users/tests/personal_preferences.rst
M plone/app/users/tests/userdata.rst
Repository: plone.app.users

Branch: refs/heads/master
Date: 2022-03-16T09:26:08+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/plone.app.users@eb12ba1

Merge pull request #107 from plone/maurits-zope-component-5

Fixed tests when run with zope.component 5+.

Files changed:
A news/500.bugfix
M plone/app/users/tests/password.rst
M plone/app/users/tests/personal_preferences.rst
M plone/app/users/tests/userdata.rst
  • Loading branch information
mauritsvanrees committed Mar 16, 2022
1 parent 7d39a2d commit 66ba940
Showing 1 changed file with 108 additions and 12 deletions.
120 changes: 108 additions & 12 deletions last_commit.txt
Original file line number Diff line number Diff line change
@@ -1,36 +1,132 @@
Repository: plone.app.customerize
Repository: plone.app.users


Branch: refs/heads/master
Date: 2022-03-08T20:59:56+01:00
Date: 2022-03-08T20:46:41+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: https://github.com/plone/plone.app.customerize/commit/dd920548cfaba228a292f4bcedccbbec14f2ece0
Commit: https://github.com/plone/plone.app.users/commit/e99f375aaa640e50d22bdd1e27357ec8fe025adc

Fixed tests when run with zope.component 5+.

See https://github.com/plone/plone.app.users/pull/107
Sample test failure:

```
File "/Users/maurits/community/plone-coredev/6.0/src/plone.app.users/plone/app/users/tests/password.rst", line 27, in password.rst
Failed example:
browser.open('http://nohost/plone/' + view_name)
Expected:
Traceback (most recent call last):
...
zExceptions.unauthorized.Unauthorized: ...You are not authorized to access this resource...
Got:
Traceback (most recent call last):
File "/Users/maurits/.pyenv/versions/3.9.9/lib/python3.9/doctest.py", line 1334, in __run
exec(compile(example.source, filename, "single",
File "&lt;doctest password.rst[8]&gt;", line 1, in &lt;module&gt;
browser.open('http://nohost/plone/' + view_name)
File "/Users/maurits/shared-eggs/cp39/zope.testbrowser-5.5.1-py3.9.egg/zope/testbrowser/browser.py", line 256, in open
self._processRequest(url, make_request)
File "/Users/maurits/shared-eggs/cp39/zope.testbrowser-5.5.1-py3.9.egg/zope/testbrowser/browser.py", line 282, in _processRequest
resp = make_request(reqargs)
File "/Users/maurits/shared-eggs/cp39/zope.testbrowser-5.5.1-py3.9.egg/zope/testbrowser/browser.py", line 253, in make_request
return self.testapp.get(url, **args)
File "/Users/maurits/shared-eggs/cp39/WebTest-3.0.0-py3.9.egg/webtest/app.py", line 324, in get
return self.do_request(req, status=status,
File "/Users/maurits/shared-eggs/cp39/zope.testbrowser-5.5.1-py3.9.egg/zope/testbrowser/browser.py", line 93, in do_request
response = super(TestbrowserApp, self).do_request(req, status,
File "/Users/maurits/shared-eggs/cp39/WebTest-3.0.0-py3.9.egg/webtest/app.py", line 620, in do_request
res = req.get_response(app, catch_exc_info=True)
File "/Users/maurits/shared-eggs/cp39/WebOb-1.8.7-py3.9.egg/webob/request.py", line 1309, in send
status, headers, app_iter, exc_info = self.call_application(
File "/Users/maurits/shared-eggs/cp39/WebOb-1.8.7-py3.9.egg/webob/request.py", line 1278, in call_application
app_iter = application(self.environ, start_response)
File "/Users/maurits/shared-eggs/cp39/WebTest-3.0.0-py3.9.egg/webtest/lint.py", line 196, in lint_app
iterator = application(environ, start_response_wrapper)
File "/Users/maurits/shared-eggs/cp39/plone.testing-8.0.3-py3.9.egg/plone/testing/_z2_testbrowser.py", line 39, in wrapped_func
return func(*args, **kw)
File "/Users/maurits/shared-eggs/cp39/plone.testing-8.0.3-py3.9.egg/plone/testing/_z2_testbrowser.py", line 66, in __call__
wsgi_result = publish(environ, start_response)
File "/Users/maurits/community/plone-coredev/6.0/src/Zope/src/ZPublisher/WSGIPublisher.py", line 391, in publish_module
request.close()
File "/Users/maurits/community/plone-coredev/6.0/src/Zope/src/ZPublisher/BaseRequest.py", line 215, in close
notify(EndRequestEvent(None, self))
File "/Users/maurits/shared-eggs/cp39/zope.event-4.5.0-py3.9.egg/zope/event/__init__.py", line 32, in notify
subscriber(event)
File "/Users/maurits/shared-eggs/cp39/zope.component-5.0.1-py3.9.egg/zope/component/event.py", line 27, in dispatch
component_subscribers(event, None)
File "/Users/maurits/shared-eggs/cp39/zope.component-5.0.1-py3.9.egg/zope/component/_api.py", line 134, in subscribers
return sitemanager.subscribers(objects, interface)
File "/Users/maurits/shared-eggs/cp39/zope.interface-5.4.0-py3.9-macosx-10.15-x86_64.egg/zope/interface/registry.py", line 448, in subscribers
return self.adapters.subscribers(objects, provided)
File "/Users/maurits/shared-eggs/cp39/zope.interface-5.4.0-py3.9-macosx-10.15-x86_64.egg/zope/interface/adapter.py", line 895, in subscribers
subscriptions = self.subscriptions([providedBy(o) for o in objects], provided)
File "/Users/maurits/shared-eggs/cp39/zope.interface-5.4.0-py3.9-macosx-10.15-x86_64.egg/zope/interface/adapter.py", line 877, in _uncached_subscriptions
if order &gt;= len(byorder):
File "/Users/maurits/.pyenv/versions/3.9.9/lib/python3.9/collections/__init__.py", line 1169, in __len__
return len(self.data)
File "/Users/maurits/shared-eggs/cp39/ZODB-5.6.0-py3.9.egg/ZODB/Connection.py", line 785, in setstate
raise ConnectionStateError(msg)
ZODB.POSException.ConnectionStateError: Shouldn't load state for persistent.list.PersistentList 0x3953f62e87808eb2 when the connection is closed
```

Should be somehow caused by this zope.component PR:
https://github.com/zopefoundation/zope.component/issues/51

From what I can follow from the traceback and running around in the pdb, it goes wrong in this part of `ZPublisher/BaseRequest.py`:

```
def close(self):
try:
notify(EndRequestEvent(None, self))
finally:
# subscribers might need the zodb, so `clear` must come afterwards
# (since `self._held=None` might close the connection, see above)
self.clear()
```

The notify fails, even when I comment out the two subscribers that I found (`Products.Five` and `plone.app.contentrules`).
Probably some other code has already called `self.clear()` before this.
A reason for this can be that several subrequests are made for the Diazo theme, and this seems to call `self.clear()` directly.
Whatever the reason, the comment is true: subscribers might need the zodb.
In fact, with zope.component 5, the list of subscribers itself, even when it is empty, is something that we get from the zodb.
So it fails here.

When I run `bin/instance fg` and visit the same page anonymously (`@@change-password`) it works fine:
I am Unauthorized and get redirected, which is also what the test expects.
So in practice all seems fine.

So what somehow triggers the ZODB error, is that this is a test, and the first browser visit is to a page that raises an Unauthorized.
The fix/workaround is to first visit the root of the site. Then somehow all is well.

Note: last time I tried this, most tests in most packages passed just fine.
`plone.app.users` was an exception, and there were about two other packages I think.
I will follow up with more PRs, but this one should be fine, also for use with our current outdated zope.component 4.6.2.

Files changed:
A news/500.bugfix
M plone/app/customerize/tests/testBrowserLayers.txt
M plone/app/users/tests/password.rst
M plone/app/users/tests/personal_preferences.rst
M plone/app/users/tests/userdata.rst

b"diff --git a/news/500.bugfix b/news/500.bugfix\nnew file mode 100644\nindex 0000000..c9f39b6\n--- /dev/null\n+++ b/news/500.bugfix\n@@ -0,0 +1,2 @@\n+Fixed tests when run with ``zope.component`` 5+.\n+[maurits]\ndiff --git a/plone/app/customerize/tests/testBrowserLayers.txt b/plone/app/customerize/tests/testBrowserLayers.txt\nindex 2077917..095e8c4 100644\n--- a/plone/app/customerize/tests/testBrowserLayers.txt\n+++ b/plone/app/customerize/tests/testBrowserLayers.txt\n@@ -20,6 +20,10 @@ product is installed, we cannot view the view, though:\n >>> browser.handleErrors = False\n >>> browser.addHeader('Authorization', 'Basic %s:%s' % (\n ... SITE_OWNER_NAME, SITE_OWNER_PASSWORD))\n+\n+First open the site root. This avoids a crazy testing-only error with zope.component 5.\n+\n+ >>> browser.open('http://nohost/plone/')\n >>> browser.open('http://nohost/plone/@@layer-test-view')\n Traceback (most recent call last):\n ...\n"
b"diff --git a/news/500.bugfix b/news/500.bugfix\nnew file mode 100644\nindex 0000000..c9f39b6\n--- /dev/null\n+++ b/news/500.bugfix\n@@ -0,0 +1,2 @@\n+Fixed tests when run with ``zope.component`` 5+.\n+[maurits]\ndiff --git a/plone/app/users/tests/password.rst b/plone/app/users/tests/password.rst\nindex 9edd74b..4b59f17 100644\n--- a/plone/app/users/tests/password.rst\n+++ b/plone/app/users/tests/password.rst\n@@ -19,6 +19,12 @@ Set up\n >>> browser = Browser(app)\n >>> browser.handleErrors = False\n \n+We must first view the homepage, otherwise we get a weird error with zope.component 5+:\n+ZODB.POSException.ConnectionStateError: Shouldn't load state for persistent.list.PersistentList 0x... when the connection is closed\n+\n+ >>> browser.open('http://nohost/plone/')\n+\n+\n The view\n ========\n \ndiff --git a/plone/app/users/tests/personal_preferences.rst b/plone/app/users/tests/personal_preferences.rst\nindex d5e88ef..b0d3fe0 100644\n--- a/plone/app/users/tests/personal_preferences.rst\n+++ b/plone/app/users/tests/personal_preferences.rst\n@@ -24,6 +24,11 @@ Set up\n ... if len(v) != 1: return False\n ... return v[0] == empty_marker\n \n+We must first view the homepage, otherwise we get a weird error with zope.component 5+:\n+ZODB.POSException.ConnectionStateError: Shouldn't load state for persistent.list.PersistentList 0x... when the connection is closed\n+\n+ >>> browser.open('http://nohost/plone/')\n+\n Viewing the personal preferences\n --------------------------------\n \ndiff --git a/plone/app/users/tests/userdata.rst b/plone/app/users/tests/userdata.rst\nindex 92b1bc3..2449697 100644\n--- a/plone/app/users/tests/userdata.rst\n+++ b/plone/app/users/tests/userdata.rst\n@@ -19,6 +19,12 @@ Set Up\n >>> browser = Browser(app)\n >>> browser.handleErrors = False\n \n+We must first view the homepage, otherwise we get a weird error with zope.component 5+:\n+ZODB.POSException.ConnectionStateError: Shouldn't load state for persistent.list.PersistentList 0x... when the connection is closed\n+\n+ >>> browser.open('http://nohost/plone/')\n+\n+\n Viewing the personal information\n ---------------------------------\n \n"

Repository: plone.app.customerize
Repository: plone.app.users


Branch: refs/heads/master
Date: 2022-03-16T09:25:59+01:00
Date: 2022-03-16T09:26:08+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: https://github.com/plone/plone.app.customerize/commit/71f66fbf604928be16f627d43ec265f91e5e6218
Commit: https://github.com/plone/plone.app.users/commit/eb12ba1071b6c301389794f0404e4c91989ceb3f

Merge pull request #20 from plone/maurits-zope-component-5
Merge pull request #107 from plone/maurits-zope-component-5

Fixed tests when run with zope.component 5+.

Files changed:
A news/500.bugfix
M plone/app/customerize/tests/testBrowserLayers.txt
M plone/app/users/tests/password.rst
M plone/app/users/tests/personal_preferences.rst
M plone/app/users/tests/userdata.rst

b"diff --git a/news/500.bugfix b/news/500.bugfix\nnew file mode 100644\nindex 0000000..c9f39b6\n--- /dev/null\n+++ b/news/500.bugfix\n@@ -0,0 +1,2 @@\n+Fixed tests when run with ``zope.component`` 5+.\n+[maurits]\ndiff --git a/plone/app/customerize/tests/testBrowserLayers.txt b/plone/app/customerize/tests/testBrowserLayers.txt\nindex 2077917..095e8c4 100644\n--- a/plone/app/customerize/tests/testBrowserLayers.txt\n+++ b/plone/app/customerize/tests/testBrowserLayers.txt\n@@ -20,6 +20,10 @@ product is installed, we cannot view the view, though:\n >>> browser.handleErrors = False\n >>> browser.addHeader('Authorization', 'Basic %s:%s' % (\n ... SITE_OWNER_NAME, SITE_OWNER_PASSWORD))\n+\n+First open the site root. This avoids a crazy testing-only error with zope.component 5.\n+\n+ >>> browser.open('http://nohost/plone/')\n >>> browser.open('http://nohost/plone/@@layer-test-view')\n Traceback (most recent call last):\n ...\n"
b"diff --git a/news/500.bugfix b/news/500.bugfix\nnew file mode 100644\nindex 0000000..c9f39b6\n--- /dev/null\n+++ b/news/500.bugfix\n@@ -0,0 +1,2 @@\n+Fixed tests when run with ``zope.component`` 5+.\n+[maurits]\ndiff --git a/plone/app/users/tests/password.rst b/plone/app/users/tests/password.rst\nindex 9edd74b..4b59f17 100644\n--- a/plone/app/users/tests/password.rst\n+++ b/plone/app/users/tests/password.rst\n@@ -19,6 +19,12 @@ Set up\n >>> browser = Browser(app)\n >>> browser.handleErrors = False\n \n+We must first view the homepage, otherwise we get a weird error with zope.component 5+:\n+ZODB.POSException.ConnectionStateError: Shouldn't load state for persistent.list.PersistentList 0x... when the connection is closed\n+\n+ >>> browser.open('http://nohost/plone/')\n+\n+\n The view\n ========\n \ndiff --git a/plone/app/users/tests/personal_preferences.rst b/plone/app/users/tests/personal_preferences.rst\nindex d5e88ef..b0d3fe0 100644\n--- a/plone/app/users/tests/personal_preferences.rst\n+++ b/plone/app/users/tests/personal_preferences.rst\n@@ -24,6 +24,11 @@ Set up\n ... if len(v) != 1: return False\n ... return v[0] == empty_marker\n \n+We must first view the homepage, otherwise we get a weird error with zope.component 5+:\n+ZODB.POSException.ConnectionStateError: Shouldn't load state for persistent.list.PersistentList 0x... when the connection is closed\n+\n+ >>> browser.open('http://nohost/plone/')\n+\n Viewing the personal preferences\n --------------------------------\n \ndiff --git a/plone/app/users/tests/userdata.rst b/plone/app/users/tests/userdata.rst\nindex 92b1bc3..2449697 100644\n--- a/plone/app/users/tests/userdata.rst\n+++ b/plone/app/users/tests/userdata.rst\n@@ -19,6 +19,12 @@ Set Up\n >>> browser = Browser(app)\n >>> browser.handleErrors = False\n \n+We must first view the homepage, otherwise we get a weird error with zope.component 5+:\n+ZODB.POSException.ConnectionStateError: Shouldn't load state for persistent.list.PersistentList 0x... when the connection is closed\n+\n+ >>> browser.open('http://nohost/plone/')\n+\n+\n Viewing the personal information\n ---------------------------------\n \n"

0 comments on commit 66ba940

Please sign in to comment.