Can't execute multiprocessing tasks using wrapt.decorator functions - not pickle serializable #158
Comments
|
I thought a workaround to the problem, if it is too hard to fix this, would be to disable the decorator and the following workaround works if the value passed to then the below won't code throw any error however - to dynamically set I see on other issue threads (mainly #102) that perhaps an ObjectProxy is picklable/serializable if we explicitly define the |
|
For dynamic function to specify whether enabled, should be: Not sure if you just cut and paste the wrong thing. But yes, it may not work as the function wrapper is still present when disabled using a function call, as is only evaluated at the time of the call. For the decorator to be applied at all, can only supply literal value. So call has to be evaluated at time of code import. Anyway, I will think about pickle issue. For this narrow case of a function wrapper decorator (as opposed to general case of object proxy), there may be a way to get it to work. Will need some investigation though. The analysis was never done for dill since that was a third party package, and so not necessarily commonly used. |
|
Sorry yes I had a typo there - just edited/corrected it ...
This would be extremely helpful - thank you for this - please let me know if I can help in the process in any way. |
|
Hi @marwan116 have you given this any more thought? |
|
The error I'm getting on python 3.11 is that Or just add the following to it: def __reduce_ex__( self, protocol ):
return (
object.__new__,
(type(self),),
object.__getstate__(self),
) |
|
@twiddli I would need to see an actual small code example of what you are trying to do to suggest anything including any explanation if it can already be done as not sure what I am trying to suggest a modification to. FWIW. Trying to pickle code is not generally a great idea. And there is no generic single |
|
Hello hello, I've reproduced this error and it very much blocks current development work using wrapt decorators. Here's the simplest setup:
from functools import wraps
from typing import Any
import wrapt
from joblib import Parallel, delayed
@wrapt.decorator
def wrapt_decorator(wrapped, instance, args, kwargs):
return wrapped(*args, **kwargs)
def functools_decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
@wrapt_decorator
def noop_func(x: Any) -> Any:
return x
if __name__ == "__main__":
with Parallel(n_jobs=2, prefer="processes") as parallel:
fs = parallel(delayed(noop_func)(x=x) for x in [1, 2, 3])
for f in fs:
print(f)Run this file - it will execute just fine.
from joblib import Parallel, delayed
from test import noop_func
if __name__ == "__main__":
with Parallel(n_jobs=2, prefer="processes") as parallel:
fs = parallel(delayed(noop_func)(x=x) for x in [1, 2, 3])
for f in fs:
print(f)Run this file and it will fail with the same above pickling error.
from typing import Any
from joblib import Parallel, delayed
def noop_func_2(x: Any) -> Any:
from test import noop_func
return noop_func(x)
if __name__ == "__main__":
with Parallel(n_jobs=2, prefer="processes") as parallel:
fs = parallel(delayed(noop_func_2)(x=x) for x in [1, 2, 3])
for f in fs:
print(f)This, once again, runs fine. Switching from How do I resolve this to make case 2 work? Is there a way to patch-through the Though this is a trivial case, this is preventing me from using a wrapt decorator in a much more complex distributed project which uses Ray and Daft. |
|
I also see that the error here makes it difficult to discern which property is raising it: Line 441 in 563525d |
|
Typo in exception strings fixed in 0da4ba5. Thanks for highlighting that one. Let me see I can get a skeleton together for doing a custom function wrapper which would allow you to override the behaviour of |
|
I think what's missing is simply a passthrough to the properties of the function. I believe that python will throw the appropriate errors if the method is not valid/callable for the underlying wrapped object. Ie. When using the functools wrapper, the multiprocessing call still runs as expected. The issue (I think) is that the |
|
Never mind - I tried patching this on my installed module and setting Why does this work perfectly with functools.wraps? And only fail with wrapt when importing across files, outside of the context of the child process? Perhaps I'm missing something fundamental. |
|
If you ignore all the extra special magic it does, for simple case of a decorator applied to a normal function, The This will now raise the override error message. With that as base you should now be able to play with how the special dunder methods might be implemented. Note that this example doesn't cater for decorators of instance methods or anything else where binding would occur in accessing the decorated function. In fact binding is in part why having a generic implementation for a function decorator might be troublesome. Now although you could defer the call of The reason it is a problem with wrapt is because decorators are implemented as a class, and more specifically a descriptor with binding. Using a class makes things much more complicated than decorators created using just nested functions. |
|
This helps - I will need to do some more reading on pickling in python as well. From what I'm understanding, I'll need to find a way to write a pickling operation for the decorator itself, and then defer to the underlying decorated object's dunder for the rest, is that correct? So long as reduce_ex() for the decorator behaves well, python should take care of the rest? The method may need to be customized for each specific decorator based on its various state variables, yes? |
|
If you check other open issues about pickling and wrapt you may actually find a partial example of how it may need to work at least in the simplest common cases. I can't remember right now and would also need to look back at all the related issues. |
I am posting this to share the issue I face when trying to use multiple processes with a function wrapped with a
wrapt.decorator.I am using the following libraries:
wrapt==1.12.1joblib=0.14.1python=3.7.6Please see the dummy example below to reproduce the error
So first I try the code using multithreading and it works fine
I get the following output:
Then when I try to use processes instead of threads:
I get an error mainly:
NotImplementedError: object proxy must define __reduce_ex__()below is the full traceback:
The text was updated successfully, but these errors were encountered: