-
-
Notifications
You must be signed in to change notification settings - Fork 381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Caching dynamic properties when using slots=True
.
#164
Comments
I've run into this as well. Unfortunately there's no easy solution as of yet. My advice for now is just use dict-based classes if you need cached properties. |
Can’t we implement something ourselves? Technically it shouldn’t be a big of a deal: private var + property in front of it? |
Well, yes, of course. Let's leave this open to track the issue. Consider the elegance of cached properties when applied to dict classes. Suppose you have a class The first time you do Next time you do This is really clean and lends itself well to being refactored out into a simple library, which is what we have today. |
Absolutely. I think we can simulate that using a simple placeholder class that does almost the same. It might make sense to have a blessed way for that that would also work with immutable classes even though it's a bit iffy. (not arguing, just sketching :)) |
I like where this is going! I played around a bit with a cache_property desriptor and If someone can sketch a basic structure that would work, I can put some work into a patch :-) |
fyi a class can be both slotted, and have a class Ham:
__slots__ = ["__dict__", "spam"]
def __init__(self):
self.__dict__ = {"spam": 3}
self.eggs = 1
self.spam = 2
print(f"{Ham().eggs=}, {Ham().spam=}, {Ham().__dict__=}") It seems you get the speedup for accessing slotted attributes |
class _Ham(dict):
__slots__ = ["ham", "spam", "eggs"]
def __getitem__(self, key):
return getattr(self, key)
def __setitem__(self, key, value):
return setattr(self, key, value)
class Ham:
__slots__ = ["__dict__", "spam"]
def __init__(self):
self.__dict__ = _Ham()
self.eggs = 1
self.spam = 2
print(f"{Ham().eggs=}, {Ham().spam=}, {Ham().__dict__=}") edit: ah this doesn't work python cheats and accesses the |
fwiw, |
Yeah, the problem is that it doesn't work with slotted classes: from functools import cached_property
import time
import attr
@attr.define
class A:
@cached_property
def x(self):
return 42
a = A()
print(a.x) throws: Traceback (most recent call last):
File "/Users/hynek/FOSS/attrs/t.py", line 14, in <module>
print(a.x)
File "/Users/hynek/.asdf/installs/python/3.10.0/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/functools.py", line 963, in __get__
raise TypeError(msg) from None
TypeError: No '__dict__' attribute on 'A' instance to cache 'x' property. Guess it could work if it stored the result in the decorator instead of the class? |
Does the stdlib cached property work with normal (non-attrs) slotted classes? |
nope: import functools
class Ham:
__slots__ = "eggs", "__weakref__"
def __init__(self, eggs):
self.eggs = eggs
@functools.cached_property
def x(self):
return 42
spam = Ham("example")
print(spam.eggs)
print(spam.x)
|
the cached property in the stdlib used |
Yeah this is not an attrs bug, but something where attrs might shine. |
not sure if this is entirely correct. this means that said, slotted classes can also have (slotted) attributes that are not set: >>> class C:
... __slots__ = ('a',)
...
>>> c = C()
>>> hasattr(c, 'a') # Not yet set
False
>>> c.a = 12
>>> hasattr(c, 'a') # Now it is set
True
>>> del c.a
>>> c.a # Now it it unset again
Traceback (most recent call last):
[...]
AttributeError: a however, trying to combine that with >>> import functools
>>> class C:
... __slots__ = ('a',)
... @functools.cached_property
... def a(self):
... return 123
...
Traceback (most recent call last):
[...]
ValueError: 'a' in __slots__ conflicts with class variable not sure if there's something that can be done about this 🤷 |
import functools
import attrs
@attrs.define()
class A:
__dict__: dict = attrs.field(default=attrs.Factory(dict), init=False, repr=False, eq=False)
@functools.cached_property
def x(self):
return 42
a = A()
a.__dict__ # {}
a.x # 42
a.__dict__ # {'x': 42} It works also for frozen slotted classes (replace A (possibly undesired) side effect of adding a.y = "test" In case such a side effect is undesired, one may opt to add the field |
I've been playing around with possibilities for getting some kind of cached property / lazy value on slotted (attrs) classes. For my use case, I'd prefer to avoid adding a dictionary to the class, to keep the memory usage small, so adding in It would be great if
and/or
Or perhaps just by detecting Anyway, I've come across 2 general approaches that seem promising. The first, option-A, is through
The second, option-B, is through a custom descriptor for the field:
Of these, option-A has the advantage of not adding overhead except:
Neither of which seems unreasonable. It does mean adding/modifying It also works the same way for non-slotted classes, so would be potentially easier to keep consistent. Option-B in contrast adds consistent overhead for accessing the cached value (i.e. even after calculating it), but doesn't spill any overhead or complexity onto anything other than the cached value. It doesn't work out of the box for non-slotted classes, so an alternate would need to be found in that case, but that shouldn't be too difficult. It's also closer in implementation to I'd be happy to take a stab at an MR for something like this when I get some time, if either of the approaches would be considered acceptable, and the interface can be roughly agreed. |
yeah B is too complicated and overhead's not cool. I think I could live with an option-A-based solution, but maybe start with a PoC and take it from there. A few notes:
It's an extra line, but I suspect it's a) clearer what's going on and b) less offensive to type checkers. IOW I was thinking something along the lines of: @attrs.define
class C:
z: int = attrs.field(init=False). # init=False would be optional but might mollify pyright etc
@z.cached
@property
def _z(self) -> int:
return But maybe it's possible to type an |
What are your thoughts on picking up I also realised this could actually be done outsids of
Applied before attrs:
I don't mean this as a long term solution, but it allows some exploration of the approach before touching attrs itself, and maybe unblocks others stumbling on this thread. (This version is also obviously not thread safe, and it would make sense to use an RLock in the same way as |
One quick fix that I'm happy with is allowing |
It seems that my proposed decorator only works when all the existing attributes are explicitly marked as Adding in a:
seems to resolve this, at least so long as the cached_property has a return annotation. I guess one could do:
for robustness. |
Btw, as properties /cached properties don't support slots as well, why not simply require the use of a cached property decorator that uses a alternative cache attribute name |
It is maybe naive, but for unknown reason, inheriting from a class is fixing this behavior : #!/usr/bin/env python3
from functools import cached_property
from attr import define
class Works:
pass
@define(frozen=True)
class Whatever(Works):
@cached_property
def it_works(self) -> int:
return 4
t = Whatever()
print(t.it_works)
print(t.__dict__)
print(t.__slots__)
|
@AdrienPensart well you have a |
slots only works as intended when all base classes use it as ell, else its mood and shouldn't e used to begin with ... |
Ran into this issue again, and passing @attrs.define
class Foo:
x: int
@attrs.lazy_field
@cached_property
def double_x(self) -> int:
return self.x * 2
foo = Foo(x=5)
print(foo.double_x) # 10 |
If we add another decorator, couldn't it be |
Depends on whether one wants the cached property to be thread safe However it's still going to require a rename of the field as to avoid having to deal with shadowed or hidden slot descriptors |
I found some time to put together a proof of concept of an approach to this: #1200 This works similarly to the decorator I posted above, in that it picks up use of the functools cached_decorator implementation, and converts it and the class to work in a similar way. It would be good to get some feedback on whether this seems like an acceptable approach at a high level. The advantage of basing on the functools cached property is that it will just work for most people, and it potentially doesn't require any API changes. An argument against is that it's makes the conversion hidden, which could be surprising, but pragmatically, that doesn't seem like it would be a common problem. There's also an issue with locking, because the current approach is per-class, which is unlikely to be acceptable in most cases. It would be good to get some feedback on what the best approach would be here. The options I see are:
|
Does attrs deal with locking in any other area? If so, this strategy should conform to that pattern. However, I think it does not - thus I would expect this to also not handle locking either, leaving room to grow out a full plan across all of attrs in the future consistently. Note that in Python 3.12+ the locking has been removed from cached_property, and this should not try to bring it back. In use cases that I think are most common, you'd want per-object locks, never per-class locks. And in those cases I'd expect to declare the entire attrs class that way, not just the cached_property but any custom setters as well. |
It doesn't that I could find, my concern was in matching
Ah, I hadn't realised that. I think that makes a fairly good argument in favour of leaving the locking out (along with the point about inconsistency with attrs). |
I've removed the locking, and tried to bring the MR up to contribution guidelines. I wasn't sure where the most appropriate place to document this behaviour would be. I've added a section to For hypothesis testing, I've added something to strategies, but there isn't prior provision for methods/non-attributes, so it might not be worth keeping the addition I've done. As a general note, I like this approach for being transparent to users, but if there's a worry about potential confusion, it should be straightforward to have a separate decorator that acts as a sentinel instead of using |
FYI this works without error now (Python 3.10, 3.11, 3.12; attrs 23.2.0).
However, there is a nasty little bug with AttributeError propagation. Let's start with a regular class:
Now let's use attrs with
So far so good. BUT:
This made me spend quite some time wondering where did my attribute go :) @hynek what do you think, shall I open a separate issue? |
ah yes yay this was fixed by #1200!
If this is somehow caused by us, yes. |
Hey there!
This is more of a question than an issue :-)
Some of the classes that I'm trying to migrate to attrs have attributes that are expensive to compute. These attributes are either never accessed or accessed multiple times in quick succession. In those cases, I prefer delaying the cost of computing the attribute in case it is not needed -- but I also never want to compute the attribute twice for the same instance. I use pydanny/cached-property for this, which works fine until you use
slots=True
.Any advice?
The text was updated successfully, but these errors were encountered: