Add BAR.rebuild() to fix the refcount issue, and to change datatypes.

It works by using the new methods allRegistrations() and allSubscriptions() to re-create the data in new data structures.

This makes fresh calls to subscribe() and register(). I went this way instead of trying to manually walk the data structures and create them because the logic in those methods is fully tested.

CHANGES.rst

@@ -20,6 +20,9 @@
   solve any already corrupted reference counts. See `issue 227
   <https://github.com/zopefoundation/zope.interface/issues/227>`_.
 
+- Add the method ``BaseAdapterRegistry.rebuild()``. This can be used
+  to fix the reference counting issue mentioned above, as well as to
+  update the data structures when custom data types have changed.
 
 5.2.0 (2020-11-05)
 ==================

src/zope/interface/adapter.py

@@ -52,19 +52,23 @@ class BaseAdapterRegistry(object):
     A basic implementation of the data storage and algorithms required
     for a :class:`zope.interface.interfaces.IAdapterRegistry`.
 
-    Subclasses or instances can set the following attributes (before
-    calling this object's ``__init__`` method) to control how the data
+    Subclasses can set the following attributes to control how the data
     is stored; in particular, these hooks can be helpful for ZODB
     persistence:
 
     _sequenceType = list
-      This is the type used for our top-level "byorder" sequences.
-      These are usually small (< 10) and are almost always accessed when
-      using this object, so it is rarely useful to change.
+      This is the type used for our two top-level "byorder" sequences.
+      These are usually small (< 10). Although at least one of them is
+      accessed when performing lookups or queries on this object, the other
+      is untouched. In many common scenarios, both are only required when
+      mutating registrations and subscriptions (like what
+      :meth:`zope.interface.interfaces.IComponents.registerUtility` does).
+      This use pattern makes it an ideal candidate to be a ``PersistentList``.
     _leafSequenceType = tuple
       This is the type used for the leaf sequences of subscribers.
       It could be set to a ``PersistentList`` to avoid many unnecessary data
-      loads when subscribers aren't being used.
+      loads when subscribers aren't being used. See :meth:`_addValueToLeaf`
+      and :meth:`removeValueFromLeaf` for two methods you'll want to override.
     _mappingType = dict
       This is the type used for the keyed mappings. A ``PersistentMapping``
       could be used to help reduce the number of data loads when the registry is large
@@ -78,6 +82,9 @@ class BaseAdapterRegistry(object):
       are always integers, so one might choose to use a more optimized data
       structure such as a ``OIBTree``. The same caveats apply as for ``_mappingType``.
 
+    It is possible to also set these on an instance, but because of the need to
+    potentially also override :meth:`addValueToLeaf` and :meth:`removeValueFromLeaf`,
+    this may be less useful in a persistent scenario; using a subclass is recommended.
 
     .. versionchanged:: 5.3.0
         Add support for customizing the way data
@@ -264,6 +271,44 @@ def registered(self, required, provided, name=u''):
 
         return components.get(name)
 
+    @classmethod
+    def _allKeys(cls, components, i, parent_k=()):
+        if i == 0:
+            for k, v in components.items():
+                yield parent_k + (k,), v
+        else:
+            for k, v in components.items():
+                new_parent_k = parent_k + (k,)
+                for x, y in cls._allKeys(v, i - 1, new_parent_k):
+                    yield x, y
+
+    def _all_entries(self, byorder):
+        # Locally reference the `byorder` data; it might be replaced while
+        # this method is running.
+        for i, components in enumerate(byorder):
+            # We will have *i* levels of dictionaries to go before
+            # we get to the leaf.
+            for key, value in self._allKeys(components, i + 1):
+                assert len(key) == i + 2
+                required = key[:i]
+                provided = key[-2]
+                name = key[-1]
+                yield (required, provided, name, value)
+
+    def allRegistrations(self):
+        """
+        Yields tuples ``(required, provided, name, value)`` for all
+        the registrations that this object holds.
+
+        These tuples could be passed as the arguments to the
+        :meth:`register` method on another adapter registry to
+        duplicate the registrations this object holds.
+
+        .. versionadded:: 5.3.0
+        """
+        for t in self._all_entries(self._adapters):
+            yield t
+
     def unregister(self, required, provided, name, value=None):
         required = tuple([_convert_None_to_Interface(r) for r in required])
         order = len(required)
@@ -325,7 +370,7 @@ def subscribe(self, required, provided, value):
         for k in key:
             d = components.get(k)
             if d is None:
-                d = {}
+                d = self._mappingType()
                 components[k] = d
             components = d
 
@@ -339,6 +384,21 @@ def subscribe(self, required, provided, value):
 
         self.changed(self)
 
+    def allSubscriptions(self):
+        """
+        Yields tuples ``(required, provided, value)`` for all the
+        subscribers that this object holds.
+
+        These tuples could be passed as the arguments to the
+        :meth:`subscribe` method on another adapter registry to
+        duplicate the registrations this object holds.
+
+        .. versionadded:: 5.3.0
+        """
+        for required, provided, _name, value in self._all_entries(self._subscribers):
+            for v in value:
+                yield (required, provided, v)
+
     def unsubscribe(self, required, provided, value=None):
         required = tuple([_convert_None_to_Interface(r) for r in required])
         order = len(required)
@@ -405,6 +465,52 @@ def unsubscribe(self, required, provided, value=None):
 
         self.changed(self)
 
+    def rebuild(self):
+        """
+        Rebuild (and replace) all the internal data structures of this
+        object.
+
+        This is useful, especially for persistent implementations, if
+        you suspect an issue with reference counts keeping interfaces
+        alive even though they are no longer used.
+
+        It is also useful if you or a subclass change the data types
+        (``_mappingType`` and friends) that are to be used.
+
+        This method replaces all internal data structures with new objects;
+        it specifically does not re-use any storage.
+
+        .. versionadded:: 5.3.0
+        """
+
+        # Grab the iterators, we're about to discard their data.
+        registrations = self.allRegistrations()
+        subscriptions = self.allSubscriptions()
+
+        def buffer(it):
+            # The generator doesn't actually start running until we
+            # ask for its next(), by which time the attributes will change
+            # unless we do so before calling __init__.
+            from itertools import chain
+            try:
+                first = next(it)
+            except StopIteration:
+                return iter(())
+
+            return chain((first,), it)
+
+        registrations = buffer(registrations)
+        subscriptions = buffer(subscriptions)
+
+
+        # Replace the base data structures as well as _v_lookup.
+        self.__init__(self.__bases__)
+        # Re-register everything previously registered and subscribed.
+        for args in registrations:
+            self.register(*args)
+        for args in subscriptions:
+            self.subscribe(*args)
+
     # XXX hack to fake out twisted's use of a private api.  We need to get them
     # to use the new registed method.
     def get(self, _): # pragma: no cover