Serialize/deserialize inconsistent for certain types in 0.4.0

[mrocklin]

In 0.4.0 I can serialize a type, but not deserialize it:

In [1]: import cloudpickle
In [2]: import threading
In [3]: lock = threading.Lock()
In [4]: b = cloudpickle.dumps(type(lock))
In [5]: cloudpickle.loads(b)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-d8541d586fca> in <module>()
----> 1 cloudpickle.loads(b)

AttributeError: type object 'lock' has no attribute 'acquire_lock'

In [6]: b
Out[6]: b'\x80\x02ccloudpickle.cloudpickle\n_rehydrate_skeleton_class\nq\x00(ccloudpickle.cloudpickle\n_builtin_type\nq\x01X\t\x00\x00\x00ClassTypeq\x02\x85q\x03Rq\x04X\x04\x00\x00\x00lockq\x05c__builtin__\nobject\nq\x06\x85q\x07}q\x08X\x07\x00\x00\x00__doc__q\tNs\x87q\nRq\x0b}q\x0c(X\x08\x00\x00\x00__repr__q\rc__builtin__\ngetattr\nq\x0eh\x0bh\r\x86q\x0fRq\x10X\x0c\x00\x00\x00acquire_lockq\x11h\x0eh\x0bh\x11\x86q\x12Rq\x13X\x07\x00\x00\x00acquireq\x14h\x0eh\x0bh\x14\x86q\x15Rq\x16X\x0c\x00\x00\x00release_lockq\x17h\x0eh\x0bh\x17\x86q\x18Rq\x19X\x07\x00\x00\x00releaseq\x1ah\x0eh\x0bh\x1a\x86q\x1bRq\x1cX\x0b\x00\x00\x00locked_lockq\x1dh\x0eh\x0bh\x1d\x86q\x1eRq\x1fX\x06\x00\x00\x00lockedq h\x0eh\x0bh \x86q!Rq"X\t\x00\x00\x00__enter__q#h\x0eh\x0bh#\x86q$Rq%X\x08\x00\x00\x00__exit__q&h\x0eh\x0bh&\x86q\'Rq(utR.'

[jakirkham]

Still an issue in cloudpickle version 0.5.2.

Also stealing from issue ( #101 ). Appears roundtrips on Enums also fail.

In [1]: import enum
   ...: class MyEnum(enum.Enum):
   ...:      SPAM = 'SPAM'
   ...: 
   ...: import cloudpickle
   ...: cloudpickle.loads(cloudpickle.dumps(MyEnum.SPAM))
   ...: 
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-3cc75bf49a58> in <module>()
      4 
      5 import cloudpickle
----> 6 cloudpickle.loads(cloudpickle.dumps(MyEnum.SPAM))

/zopt/conda2/envs/nanshenv3/lib/python3.6/enum.py in __new__(metacls, cls, bases, classdict)
    133         # save enum items into separate mapping so they don't get baked into
    134         # the new class
--> 135         enum_members = {k: classdict[k] for k in classdict._member_names}
    136         for name in classdict._member_names:
    137             del classdict[name]

AttributeError: 'dict' object has no attribute '_member_names'

Edit: Fixed in 1.2.1.

[pierreglaser]

the _thread.Lock builtin class is not importable via from _thread import Lock, so cloudpickle labels this class as dynamic. Pickling builtins as dynamic has happened more than once in the past, and leads to undefined behavior.

I think that for this issue in particular we should special-case _thread.Lock objects by adding a corresponding entry in dispatch
EDIT: the problem lies in serializing the class, not the object. So we cannot use the dispatch_table.

[jakirkham]

That sounds reasonable.

Stepping back a bit. It's probably fair to ask, why does one want to pickle a lock?

[pierreglaser]

Stepping back a bit. It's probably fair to ask, why does one want to pickle a lock?

Agreed, I have no idea if that even makes sense.

But note that in this setting, cloudpickle fails to serialize the Lock class, and not lock instances.

[jakirkham]

Good point. Was thrown off by the slightly circuitous nature of the code.

[ogrisel]

So what should we do? Add an entry to the cloudpickle dispatch table to make sure we raise TypeError or PicklingError at dump time (instead of AttributeError at load time)?

[ogrisel]

This might prevent pickling interactively defined functions that have reference a lock instance as global variable in __main__ for instance.