Proposal: Lazy Property

[atyutin]

This is a copy of an issue opened at enthought/traits.

I propose to add "lazy property" to the existing trait types. The purpose is to avoid unnecessary computations of property values and redundant trait notifications.

Consider first a standard property trait. Suppose that there are traits (call them “ancestors”) that the property depends on (which is the usual case for a non-trivial property), and there are also traits (“descendants”) that depend on the considered property. In short, we suppose that the property is an intermediate node in the traits notification network.

My concern is that each time such a property receives a notification that one of its ancestors has changed its value, it automatically recomputes its own value whether or not this value is going to be used.

I propose to add "lazy property" to the existing trait types. The purpose is to avoid unnecessary computations of property values and redundant trait notifications.

Consider first a standard property trait. Suppose that there are traits (call them “ancestors”) that the property depends on (which is the usual case for a non-trivial property), and there are also traits (“descendants”) that depend on the considered property. In short, we suppose that the property is an intermediate node in the traits notification network.

My concern is that each time such a property receives a notification that one of its ancestors has changed its value, it automatically recomputes its own value whether or not this value is going to be used.

In contrast, a “lazy property” should be such property that when it receives a notification, it does not update its value immediately, but postpones the updating until the value is actually requested (by a getattr method or alike). This "lazy evaluation" can be complemented with "lazy notification", which means that if the “lazy property” has already sent a notification, it does not send further notifications until its value is recomputed.

Let us now recall why the “usual” property with descendants automatically recomputes its value. This happens because the property must propagate the received notification, telling its descendants that its value is being updated, and the current implementation of Traits is such that the new value must be part of the notification. But is it really necessary to include the new value in the notification? Would it not suffice just to notify the descendants that the value is not valid anymore? More specifically, it appears to be no important reason to include both the old and the new values in the trait_change event. Posting only the old value would be sufficient, at least in most cases; the new value can be always readily computed by explicitly accessing the trait. However, to conform to the existing Traits framework, one can simply include the “Undefined” object in the trait notification instead of the real new value.

Where could "lazy properties" be useful? Clearly, the Traits properties offer an elegant syntax to implement a function as a composition of other functions (with an unlimited number of "intermediate" decomposition layers). This is useful in coding many mathematical models (in particular, in the context of Chaco visualization). But what can be said about the computational efficiency of this framework? A function coded as a
"network" of properties reacts to any change of its inputs. This is acceptable if one wishes to trace the changes in function values caused by each change in each individual input. But suppose that we don't need the function value be recomputed after each change. We may wish to change a subset of the inputs, and evaluate the function after that. Here the standard properties become inefficient, resulting in unnecessary computations and notifications. Let me refer to my code below (or script lazy_property.py) for an illustration.

This ticket follows the discussion on the Enthought-Dev mailing list (see subject [Traits] Proposal: lazy property, initiated by Anton Tyutin on February 7, 2011). You can also find a link to script lazy_property.py there, which is a simple implementation of the "lazy evaluation" feature of “lazy properties”. This code is cited below for completeness.

from __future__ import print_function

from enthought.traits.api import Undefined, Property
from enthought.traits.has_traits import weak_arg


def enable_lazy_properties(cls):
    """ Decorator that enables class cls to have 'lazy' Property traits by
        overriding method _init_trait_property_listener. The class cls is 
        expected to be a subclass of class HasTraits (defined in 
        enthought.traits.has_traits).

        A 'lazy' property is a Property trait with 'depends_on' metadata, and,
        additionally, 'lazy' metadata set to True. When a 'lazy' property 
        receives a notification from a trait it depends on, it sends the
        Undefined object as its new value to its listeners; a 'lazy' property 
        does not compute its actual value when receiving notifications, only its
        cached value is invalidated, if the latter exists.
    """

    # define new method    
    def _init_trait_property_listener ( self, name, kind, cached, pattern ):
        """ Sets up the listener for a property with 'depends_on' metadata.
        """
        property_trait = self.__class_traits__[name]
        if property_trait.__dict__.get('lazy', False):
            def my_trait_property_changed(self, name, old):
#                print(name) ## Uncomment this line if you wish to trace notifications
                return self.trait_property_changed( name, old, Undefined )
        else:
            def my_trait_property_changed(self, name, old):
                return self.trait_property_changed( name, old )

        if cached is None:
            @weak_arg(self)
            def notify ( self ):
                my_trait_property_changed(self, name, None)
        else:
            cached_old = cached + ':old'
            @weak_arg(self)
            def pre_notify ( self ):
                dict = self.__dict__
                old  = dict.get( cached_old, Undefined )
                if old is Undefined:
                    dict[ cached_old ] = dict.pop( cached, None )
            self.on_trait_change( pre_notify, pattern, priority = True, target=self )

            @weak_arg(self)
            def notify ( self ):
                old = self.__dict__.pop( cached_old, Undefined )
                if old is not Undefined:
                    my_trait_property_changed(self, name, old)

        self.on_trait_change( notify, pattern, target=self )

    # override the method
    cls._init_trait_property_listener = _init_trait_property_listener

    # return the modified class
    return cls


def LazyProperty(lazily_depends_on, *args, **kwdargs):
    """ Shortcut to a lazy property constructor.
    """
    return Property(*args, depends_on=lazily_depends_on, lazy=True, **kwdargs)


def ReadOnlyElement(prop_name, ind):
    """ 
    """
    return LazyProperty(prop_name, 
                        fget = lambda self: getattr(self, prop_name)[ind])


if __name__ == "__main__":
    # testing example

    from enthought.traits.api import HasTraits, Int, Property, cached_property

    print("\nCompute z = y * x3, where y = x1 + x2, using lazy properties:")

    @enable_lazy_properties
    class A(HasTraits):

        x1 = Int
        x2 = Int
        x3 = Int

        y = LazyProperty('x1, x2')
        @cached_property
        def _get_y(self):
            print("_get_y called")
            return self.x1 + self.x2

        z = LazyProperty('y, x3')
        @cached_property
        def _get_z(self):
            print("_get_z called")
            return self.y * self.x3

    a = A()

    print("")
    print("x1 set")
    a.x1 = 2
    print("x2 set")
    a.x2 = 3
    print("x3 set")
    a.x3 = 4
    print("z accessed")
    print("z =", a.z)

    print("")
    print("x3 set")
    a.x3 = 8
    print("z accessed")
    print("z =", a.z)

    print("")
    print("x1 set")
    a.x1 = 6
    print("x2 set")
    a.x2 = 4
    print("z accessed")
    print("z =", a.z)


    print("\n\nNow do analogous computations using traditional properties:")

#    @enable_lazy_properties  ## You may uncomment this line: nothing should change in the output
    class A0(HasTraits):

        x1 = Int
        x2 = Int
        x3 = Int

        y = Property(depends_on='x1, x2')
        @cached_property
        def _get_y(self):
            print("_get_y called")
            return self.x1 + self.x2

        z = Property(depends_on='y, x3')
        @cached_property
        def _get_z(self):
            print("_get_z called")
            return self.y * self.x3

    a = A0()

    print("")
    print("x1 set")
    a.x1 = 2
    print("x2 set")
    a.x2 = 3
    print("x3 set")
    a.x3 = 4
    print("z accessed")
    print("z =", a.z)

    print("")
    print("x3 set")
    a.x3 = 8
    print("z accessed")
    print("z =", a.z)

    print("")
    print("x1 set")
    a.x1 = 6
    print("x2 set")
    a.x2 = 4
    print("z accessed")
    print("z =", a.z)

[sccolbert]

I'm not particularly keen on the semantics of this. It seems like what you really want, is the ability to block trait change notifications temporarily while you change a bunch of traits, then send out a batch notification. Such behavior is something I think traits4 should support, but syntax will very likely not be through lazy properties (I'm thinking an event system context guard here.)

[atyutin]

Simple blocking/unblocking of notification will not help. This issue is not just about a kind of "Apply" button mechanism. For example, let we have traits x, y, and cached properties z(x,y) and f(z). We can block temporarily notifications from x and y, change them as many times as we want, then unblock the notifications, and access the value of f. However, if the batch notification is simply a series of usual notifications, the property z will be computed twice: once after receiving a notification from x, and also after receiving a notification from y. In general, it depends very much on the model of batch processing whether such redundant computations can be avoided by the blocking/unblocking mechanism.

[atyutin]

I would also like to draw attention to the question: is it really necessary to include the new value into the trait change event?

[sccolbert]

It does depend on the model of batch processing, which is why the framework in traits4 is allowing for dynamic flexibility in that, and many other areas, so that lazy patterns need not be restricted to just certain types.

To answer your question about a new value being necessary in a trait change event: yes.

[atyutin]

Thanks. I am glad to know that the problems of trait notification efficiency that I tried to point out are being solved, and that this is being done at a more fundamental level than I could imagine. I would certainly like to learn more about the new framework, and I would very much appreciate an overview of how it works.

[bergtholdt]

Hi,

I'm actually using this recipe quite often, but now have found an issue that I cannot work around: The problem is when using LazyProperty with traitsui and defining a view that has some Item use the enabled_when/ visible_when/ checked_when system for defining visibility, etc. Then in ui.py an anytrait handler is setup that will evaluate all values of the object and also the lazy properties, which kind of defeats the purpose of my lazy properties.

Example:

from __future__ import print_function
from plus import enable_lazy_properties, LazyProperty
from traits.api import *
from traitsui.api import *


@enable_lazy_properties
class A(HasTraits):
    test_event = Event()
    a = LazyProperty('test_event')
    b = LazyProperty('test_event')
    c = LazyProperty('test_event')

    @cached_property
    def _get_a(self):
        print('a')
        return 'a'

    @cached_property
    def _get_b(self):
        print('b')
        return 'b'

    @cached_property
    def _get_c(self):
        print('c')
        return 'c'

Now using an interactive prompt (ipython) I do this:

>>> view = View(Item('a'))
>>> a = A()
>>> a.edit_traits(view=view)
>>> a.test_event = 1
a

But with enabled_when defined:

>>> view = View(Item('a', enabled_when='True'))
>>> a = A()
>>> a.edit_traits(view=view)
>>> a.test_event = 1
a
c
b
a
c
b

i.e. at the initialization of a.edit_traits and each time I set test_event, all lazy properties get updated.

It would be nice if someone could come up with a workaround.