pickling annotations / mappingproxy objects

[satra]

cloudpickle appears to still have an issue (see #193, #209) with pickling type annotations. the reason i can't use pickle is that it doesn't support dynamic dataclasses (see: https://bugs.python.org/issue35510), which cloudpickle does support.

In [1]: import dataclasses as dc                                                  
In [2]: import typing as ty                                                       
In [3]: import cloudpickle as cp                                                  
In [4]: @dc.dataclass 
   ...: class A: 
   ...:      a: ty.Optional[int]=None 
   ...:                                                                           
In [5]: cp.dumps(A)                                                               
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-63d18a4a1006> in <module>
----> 1 cp.dumps(A)

/software/miniconda3/envs/pydra-dev/lib/python3.7/site-packages/cloudpickle/cloudpickle.py in dumps(obj, protocol)
    929     try:
    930         cp = CloudPickler(file, protocol=protocol)
--> 931         cp.dump(obj)
    932         return file.getvalue()
    933     finally:

/software/miniconda3/envs/pydra-dev/lib/python3.7/site-packages/cloudpickle/cloudpickle.py in dump(self, obj)
    282         self.inject_addons()
    283         try:
--> 284             return Pickler.dump(self, obj)
    285         except RuntimeError as e:
    286             if 'recursion' in e.args[0]:

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in dump(self, obj)
    435         if self.proto >= 4:
    436             self.framer.start_framing()
--> 437         self.save(obj)
    438         self.write(STOP)
    439         self.framer.end_framing()

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506 

/software/miniconda3/envs/pydra-dev/lib/python3.7/site-packages/cloudpickle/cloudpickle.py in save_global(self, obj, name, pack)
    676 
    677         if obj.__module__ == "__main__":
--> 678             return self.save_dynamic_class(obj)
    679 
    680         try:

/software/miniconda3/envs/pydra-dev/lib/python3.7/site-packages/cloudpickle/cloudpickle.py in save_dynamic_class(self, obj)
    515         # Now save the rest of obj's __dict__. Any references to obj
    516         # encountered while saving will point to the skeleton class.
--> 517         save(clsdict)
    518 
    519         # Write a tuple of (skeleton_class, clsdict).

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506 

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save_dict(self, obj)
    854 
    855         self.memoize(obj)
--> 856         self._batch_setitems(obj.items())
    857 
    858     dispatch[dict] = save_dict

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in _batch_setitems(self, items)
    880                 for k, v in tmp:
    881                     save(k)
--> 882                     save(v)
    883                 write(SETITEMS)
    884             elif n:

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506 

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save_dict(self, obj)
    854 
    855         self.memoize(obj)
--> 856         self._batch_setitems(obj.items())
    857 
    858     dispatch[dict] = save_dict

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in _batch_setitems(self, items)
    885                 k, v = tmp[0]
    886                 save(k)
--> 887                 save(v)
    888                 write(SETITEM)
    889             # else tmp is empty, and we're done

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    547 
    548         # Save the reduce() output and finally memoize the object
--> 549         self.save_reduce(obj=obj, *rv)
    550 
    551     def persistent_id(self, obj):

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    660 
    661         if state is not None:
--> 662             save(state)
    663             write(BUILD)
    664 

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506 

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save_tuple(self, obj)
    769         if n <= 3 and self.proto >= 2:
    770             for element in obj:
--> 771                 save(element)
    772             # Subtle.  Same as in the big comment below.
    773             if id(obj) in memo:

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    502         f = self.dispatch.get(t)
    503         if f is not None:
--> 504             f(self, obj) # Call unbound method with explicit self
    505             return
    506 

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save_dict(self, obj)
    854 
    855         self.memoize(obj)
--> 856         self._batch_setitems(obj.items())
    857 
    858     dispatch[dict] = save_dict

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in _batch_setitems(self, items)
    880                 for k, v in tmp:
    881                     save(k)
--> 882                     save(v)
    883                 write(SETITEMS)
    884             elif n:

/software/miniconda3/envs/pydra-dev/lib/python3.7/pickle.py in save(self, obj, save_persistent_id)
    522             reduce = getattr(obj, "__reduce_ex__", None)
    523             if reduce is not None:
--> 524                 rv = reduce(self.proto)
    525             else:
    526                 reduce = getattr(obj, "__reduce__", None)

TypeError: can't pickle mappingproxy objects

[ogrisel]

I think this is in the scope of cloudpickle however I have no idea about the details of how it should be implemented. If you want to give it a try, please feel free to submit a PR.

[satra]

@ogrisel - i don't know if i have a solution to this at this point. i ended up using a mixture of pickle and cloudpickle i think. let's leave this open for now, and see if someone else pitches in.

[pcmoritz]

this might be fixed in #245

[JessicaSchrouff]

A fix would be great as we are running into similar issues.
Thank you for the help!

[ogrisel]

#245 has been merged in master.

[ogrisel]

The fix to support dataclasses is now available in cloudpickle 0.8.0 on PyPI.