Proposal: Lazy Property

[atyutin]

I propose to add "lazy property" to the existing trait types. The purpose is to avoid unnecessary computations of property values and redundant trait notifications.

Consider first a standard property trait. Suppose that there are traits (call them “ancestors”) that the property depends on (which is the usual case for a non-trivial property), and there are also traits (“descendants”) that depend on the considered property. In short, we suppose that the property is an intermediate node in the traits notification network.

My concern is that each time such a property receives a notification that one of its ancestors has changed its value, it automatically recomputes its own value whether or not this value is going to be used.

I propose to add "lazy property" to the existing trait types. The purpose is to avoid unnecessary computations of property values and redundant trait notifications.

Consider first a standard property trait. Suppose that there are traits (call them “ancestors”) that the property depends on (which is the usual case for a non-trivial property), and there are also traits (“descendants”) that depend on the considered property. In short, we suppose that the property is an intermediate node in the traits notification network.

My concern is that each time such a property receives a notification that one of its ancestors has changed its value, it automatically recomputes its own value whether or not this value is going to be used.

In contrast, a “lazy property” should be such property that when it receives a notification, it does not update its value immediately, but postpones the updating until the value is actually requested (by a getattr method or alike). This "lazy evaluation" can be complemented with "lazy notification", which means that if the “lazy property” has already sent a notification, it does not send further notifications until its value is recomputed.

Let us now recall why the “usual” property with descendants automatically recomputes its value. This happens because the property must propagate the received notification, telling its descendants that its value is being updated, and the current implementation of Traits is such that the new value must be part of the notification. But is it really necessary to include the new value in the notification? Would it not suffice just to notify the descendants that the value is not valid anymore? More specifically, it appears to be no important reason to include both the old and the new values in the trait_change event. Posting only the old value would be sufficient, at least in most cases; the new value can be always readily computed by explicitly accessing the trait. However, to conform to the existing Traits framework, one can simply include the “Undefined” object in the trait notification instead of the real new value.

Where could "lazy properties" be useful? Clearly, the Traits properties offer an elegant syntax to implement a function as a composition of other functions (with an unlimited number of "intermediate" decomposition layers). This is useful in coding many mathematical models (in particular, in the context of Chaco visualization). But what can be said about the computational efficiency of this framework? A function coded as a
"network" of properties reacts to any change of its inputs. This is acceptable if one wishes to trace the changes in function values caused by each change in each individual input. But suppose that we don't need the function value be recomputed after each change. We may wish to change a subset of the inputs, and evaluate the function after that. Here the standard properties become inefficient, resulting in unnecessary computations and notifications. Let me refer to my code below (or script lazy_property.py) for an illustration.

This ticket follows the discussion on the Enthought-Dev mailing list (see subject [Traits] Proposal: lazy property, initiated by Anton Tyutin on February 7, 2011). You can also find a link to script lazy_property.py there, which is a simple implementation of the "lazy evaluation" feature of “lazy properties”. This code is cited below for completeness.

from __future__ import print_function

from enthought.traits.api import Undefined, Property
from enthought.traits.has_traits import weak_arg


def enable_lazy_properties(cls):
    """ Decorator that enables class cls to have 'lazy' Property traits by
        overriding method _init_trait_property_listener. The class cls is 
        expected to be a subclass of class HasTraits (defined in 
        enthought.traits.has_traits).

        A 'lazy' property is a Property trait with 'depends_on' metadata, and,
        additionally, 'lazy' metadata set to True. When a 'lazy' property 
        receives a notification from a trait it depends on, it sends the
        Undefined object as its new value to its listeners; a 'lazy' property 
        does not compute its actual value when receiving notifications, only its
        cached value is invalidated, if the latter exists.
    """

    # define new method    
    def _init_trait_property_listener ( self, name, kind, cached, pattern ):
        """ Sets up the listener for a property with 'depends_on' metadata.
        """
        property_trait = self.__class_traits__[name]
        if property_trait.__dict__.get('lazy', False):
            def my_trait_property_changed(self, name, old):
#                print(name) ## Uncomment this line if you wish to trace notifications
                return self.trait_property_changed( name, old, Undefined )
        else:
            def my_trait_property_changed(self, name, old):
                return self.trait_property_changed( name, old )

        if cached is None:
            @weak_arg(self)
            def notify ( self ):
                my_trait_property_changed(self, name, None)
        else:
            cached_old = cached + ':old'
            @weak_arg(self)
            def pre_notify ( self ):
                dict = self.__dict__
                old  = dict.get( cached_old, Undefined )
                if old is Undefined:
                    dict[ cached_old ] = dict.pop( cached, None )
            self.on_trait_change( pre_notify, pattern, priority = True, target=self )

            @weak_arg(self)
            def notify ( self ):
                old = self.__dict__.pop( cached_old, Undefined )
                if old is not Undefined:
                    my_trait_property_changed(self, name, old)

        self.on_trait_change( notify, pattern, target=self )

    # override the method
    cls._init_trait_property_listener = _init_trait_property_listener

    # return the modified class
    return cls


def LazyProperty(lazily_depends_on, *args, **kwdargs):
    """ Shortcut to a lazy property constructor.
    """
    return Property(*args, depends_on=lazily_depends_on, lazy=True, **kwdargs)


def ReadOnlyElement(prop_name, ind):
    """ 
    """
    return LazyProperty(prop_name, 
                        fget = lambda self: getattr(self, prop_name)[ind])


if __name__ == "__main__":
    # testing example

    from enthought.traits.api import HasTraits, Int, Property, cached_property

    print("\nCompute z = y * x3, where y = x1 + x2, using lazy properties:")

    @enable_lazy_properties
    class A(HasTraits):

        x1 = Int
        x2 = Int
        x3 = Int

        y = LazyProperty('x1, x2')
        @cached_property
        def _get_y(self):
            print("_get_y called")
            return self.x1 + self.x2

        z = LazyProperty('y, x3')
        @cached_property
        def _get_z(self):
            print("_get_z called")
            return self.y * self.x3

    a = A()

    print("")
    print("x1 set")
    a.x1 = 2
    print("x2 set")
    a.x2 = 3
    print("x3 set")
    a.x3 = 4
    print("z accessed")
    print("z =", a.z)

    print("")
    print("x3 set")
    a.x3 = 8
    print("z accessed")
    print("z =", a.z)

    print("")
    print("x1 set")
    a.x1 = 6
    print("x2 set")
    a.x2 = 4
    print("z accessed")
    print("z =", a.z)


    print("\n\nNow do analogous computations using traditional properties:")

#    @enable_lazy_properties  ## You may uncomment this line: nothing should change in the output
    class A0(HasTraits):

        x1 = Int
        x2 = Int
        x3 = Int

        y = Property(depends_on='x1, x2')
        @cached_property
        def _get_y(self):
            print("_get_y called")
            return self.x1 + self.x2

        z = Property(depends_on='y, x3')
        @cached_property
        def _get_z(self):
            print("_get_z called")
            return self.y * self.x3

    a = A0()

    print("")
    print("x1 set")
    a.x1 = 2
    print("x2 set")
    a.x2 = 3
    print("x3 set")
    a.x3 = 4
    print("z accessed")
    print("z =", a.z)

    print("")
    print("x3 set")
    a.x3 = 8
    print("z accessed")
    print("z =", a.z)

    print("")
    print("x1 set")
    a.x1 = 6
    print("x2 set")
    a.x2 = 4
    print("z accessed")
    print("z =", a.z)

[bergtholdt]

Hi,

what ever happened to this? I just wanted to implement the same "pull"-driven behavior, tried to implement it using cached properties and stumbled upon the same issue of intermediate properties where updates are always triggered. I like this solution and at first glance seems to work for me. Is there any other more "official" way in traits 4.1 on how to implement that "lazy evaluation" pattern?

[atyutin]

Hi,

I am pleased to know that someone has got seriously interested in this topic. I decided to stop promoting the idea after sccolbert mentioned that some mechanism implementing the discussed lazy evaluation pattern could be included in traits 4, please see comments on enthought/traits4#1. Unfortunately, I don't know if anything like that was really implemented.

[bergtholdt]

Hi,

indeed I read the comments, but could not find any hints on how this changed with Traits 4.0. Maybe I need to look for a changelog to traits 4...