[Ray 2.3 release] [data] read_parquet_benchmark_single_node release test failure

[cadedaniel]

https://buildkite.com/ray-project/release-tests-branch/builds/1303#0185d157-4e6a-4e4e-9629-8badcc674ecc

Traceback (most recent call last):
  File "read_parquet_benchmark.py", line 88, in <module>
    run_read_parquet_benchmark(benchmark)
  File "read_parquet_benchmark.py", line 54, in run_read_parquet_benchmark
    filter=filter_expr,
  File "/home/ray/oss-release-tests-2/benchmark.py", line 49, in run
    output_ds = fn(**fn_run_args)
  File "read_parquet_benchmark.py", line 25, in read_parquet
    columns=columns,
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/data/read_api.py", line 499, in read_parquet
    **arrow_parquet_args,
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/data/read_api.py", line 311, in read_datasource
    _wrap_arrow_serialization_workaround(read_args),
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/remote_function.py", line 129, in _remote_proxy
    return self._remote(args=args, kwargs=kwargs, **self._default_options)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 307, in _invocation_remote_span
    return method(self, args, kwargs, *_args, **_kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/remote_function.py", line 412, in _remote
    return invocation(args, kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/remote_function.py", line 399, in invocation
    serialized_runtime_env_info or "{}",
  File "python/ray/_raylet.pyx", line 1968, in ray._raylet.CoreWorker.submit_task
  File "python/ray/_raylet.pyx", line 1972, in ray._raylet.CoreWorker.submit_task
  File "python/ray/_raylet.pyx", line 424, in ray._raylet.prepare_args_and_increment_put_refs
  File "python/ray/_raylet.pyx", line 415, in ray._raylet.prepare_args_and_increment_put_refs
  File "python/ray/_raylet.pyx", line 462, in ray._raylet.prepare_args_internal
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/serialization.py", line 450, in serialize
    return self._serialize_to_msgpack(value)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/serialization.py", line 429, in _serialize_to_msgpack
    metadata, python_objects
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/serialization.py", line 390, in _serialize_to_pickle5
    raise e
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/serialization.py", line 386, in _serialize_to_pickle5
    value, protocol=5, buffer_callback=writer.buffer_callback
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 88, in dumps
    cp.dump(obj)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 733, in dump
    return Pickler.dump(self, obj)
  File "pyarrow/io.pxi", line 1090, in pyarrow.lib.Buffer.__reduce_ex__
AttributeError: module 'pickle' has no attribute 'PickleBuffer'

[scottjlee]

I think from previous discussion, we determined this could potentially be a Core issue.

@stephanie-wang I think you mentioned last week you could look into this, but I know you're out this week -- let me know if you want me to take this on instead

In my initial investigation previously, I found 1 and 2 suggesting that the fix is upgrading to Python 3.8, not sure if it's the same case here.

[cadedaniel]

@scottjlee do we have a repro script to see if it's a core issue?

[scottjlee]

Actually, looks like there was a similar issue here, where the solution was to install pickle5 for Python 3.7: #26225

If I'm understanding correctly, do I simply need to add the pickle5 package here, in the test's app_config.yaml?