More robust pickling support

[bluescarni]

The current documentation about pickling support states:

Note that only the cPickle module is supported on Python 2.7. It is also important to request usage of the highest protocol version using the -1 argument to dumps. Failure to follow these two steps will lead to important pybind11 memory allocation routines to be skipped during unpickling, which will likely cause memory corruption and/or segmentation faults.

The cPickle restriction for Python 2.x does not look too problematic, but I believe the requirement about the explicit protocol version is an issue. Forcing the -1 argument in internal use is not problematic if one writes a couple of thin wrappers around loads/dumps for the purpose, but it is fragile with respect to those situations in which one cannot control how loads/dumps are called (e.g., the multiprocessing module, ipyparallel, etc.).

Would it be possible to remove this restriction?

[wjakob]

I don't think you absolutely need the latest version. It's just that the default (oldest) one doesn't invoke all the callbacks in a way that is needed for pybind11, which causes a crash.

[bluescarni]

@wjakob Thanks for the quick reply!

So do you think it would be ok in general (let's say Python 2.7 and 3.x) to stick with the default protocol version?

[wjakob]

Try it yourself -- I think it crashes with the default.

[wjakob]

So the minimum version turns out to be 2, which was introduced for new-style classes. All of this works fine with multiprocessing by the way, it seems to use >= 2.

[bluescarni]

@wjakob Thanks a lot for looking into this!