cloudpickle.dumps() uses protocol 2 by default

[pitrou]

I'm not sure why that is, but protocol 3 should be the default on Python 3 and can be significantly faster:

>>> b = b"x" * (1024**2)
>>> %timeit cloudpickle.dumps(b)
1.45 ms ± 5.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>>> %timeit cloudpickle.dumps(b, protocol=3)
586 µs ± 2.67 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Unpickling is also faster:

>>> d1 = cloudpickle.dumps(b)
>>> d2 = cloudpickle.dumps(b, protocol=3)
>>> %timeit pickle.loads(d1)
225 µs ± 843 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>>> %timeit pickle.loads(d2)
67.1 µs ± 348 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

[mrocklin]

I suspect that this is so that Python 3 pickled bytes can sometimes be unpickled on Python 2. I'm not at all defending this decision, just posing a guess.

[mrocklin]

@pitrou , why not HIGHEST_PROTOCOL?

[pitrou]

Standard pickle uses DEFAULT_PROTOCOL, IMHO it's less confusing if we don't deviate from it. But no strong opinion either.

[ogrisel]

I think the main use case for cloudpickle is to allow interprocess communication between Python workers in a parallel computing system (PySpark, dask, joblib). Pickled objects will probably not be stored in the long term, just the time of sending a message to the next worker and we can reasonably expect that all the Python processes communicating together in such a system will have the same version so using HIGHEST_PROTOCOL should be fine in that context.

However, some users might already use it in another scenario where Python 2 / Python 3 communication is actually a use case.

[rgbkrk]

Pickled objects will probably not be stored in the long term, just the time of sending a message to the next python process and we can reasonably expect that all the Python processes communicating together in such a system will have the same version so using HIGHEST_PROTOCOL should be fine in that context.

Totally in agreement with this. I think we should switch to HIGHEST_PROTOCOL by default. For the users I support, their primary use is on Spark. I'd prefer the performance over potential Python process interop. As long as the drivers and the executors match, I think we're ok.

@rdblue -- when people run our Python3 kernel with Spark, does the executor match the driver?

[HyukjinKwon]

Hi all, I happen to come here while reading the recent interesting issues and changes.

Here are Spark related JIRA and PR, https://issues.apache.org/jira/browse/SPARK-18161 and https://github.com/apache/spark/pull/15670. I hope this could be helpful.

[rdblue]

Yes, we always start executors with the same Spark build that was used for the driver.

[pitrou]

Fair enough. Let's use HIGHEST_PROTOCOL then :-)

[ogrisel]

See PR #127 to implement the switch.

[Peque]

#127 was merged, so closing this...