LocalCluster crashed with: distributed.core - ERROR - 'tuple' object does not support item assignment

[felix0097]

What happened:
Python programs crashes with the following error message:

distributed.core - ERROR - 'tuple' object does not support item assignment
Traceback (most recent call last):
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 554, in handle_stream
    msgs = await comm.read()
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/inproc.py", line 199, in read
    msg = nested_deserialize(msg)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 609, in nested_deserialize
    return replace_inner(x)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 601, in replace_inner
    x[k] = replace_inner(v)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 594, in replace_inner
    x[k] = deserialize(v.header, v.frames)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 384, in deserialize
    return loads(header, frames)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 77, in pickle_loads
    buffers[i] = memoryview(bytearray(mv)).cast(mv.format, mv.shape)
TypeError: 'tuple' object does not support item assignment
distributed.worker - ERROR - 'tuple' object does not support item assignment
Traceback (most recent call last):
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 990, in handle_scheduler
    await self.handle_stream(
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 554, in handle_stream
    msgs = await comm.read()
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/inproc.py", line 199, in read
    msg = nested_deserialize(msg)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 609, in nested_deserialize
    return replace_inner(x)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 601, in replace_inner
    x[k] = replace_inner(v)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 594, in replace_inner
    x[k] = deserialize(v.header, v.frames)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 384, in deserialize
    return loads(header, frames)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 77, in pickle_loads
    buffers[i] = memoryview(bytearray(mv)).cast(mv.format, mv.shape)
TypeError: 'tuple' object does not support item assignment
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f7e6575cc40>>, <Task finished name='Task-22' coro=<Worker.handle_scheduler() done, defined at /home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py:988> exception=TypeError("'tuple' object does not support item assignment")>)
Traceback (most recent call last):
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 990, in handle_scheduler
    await self.handle_stream(
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 554, in handle_stream
    msgs = await comm.read()
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/inproc.py", line 199, in read
    msg = nested_deserialize(msg)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 609, in nested_deserialize
    return replace_inner(x)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 601, in replace_inner
    x[k] = replace_inner(v)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 594, in replace_inner
    x[k] = deserialize(v.header, v.frames)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 384, in deserialize
    return loads(header, frames)
  File "/home/ffischer/anaconda3/envs/dask/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 77, in pickle_loads
    buffers[i] = memoryview(bytearray(mv)).cast(mv.format, mv.shape)
TypeError: 'tuple' object does not support item assignment

What you expected to happen:
No error: persist or compute call returns as usual

Minimal Complete Verifiable Example:

import pandas as pd
import dask.dataframe as dd

from dask.distributed import Client, LocalCluster

# if you dont create a local cluster and the client everything works
cluster = LocalCluster(processes=False)
client = Client(cluster, set_as_default=True)

def do_nothing(df, other_stuff):
    return df

ddf = (
    dd.from_pandas(pd.DataFrame(range(20)), npartitions=5)
    .map_partitions(do_nothing, other_stuff=1, meta={0: 'i8'})
    .persist()
)

Anything else we need to know?:
Without creating the LocalCluster and Client everything works as expected, so I would guess this issue has something to do with the LocalCluster

With dask version 2021.2 everything works correctly

Environment:

• Dask version: 2021.3
• Python version: 3.8.8
• Operating System: Ubuntu 20.04.2 LTS
• Install method (conda, pip, source): conda install dask=2021.3

[jrbourbeau]

Thanks for reporting and providing a nice minimal example @felix0097! I'm able to reproduce on the latest main branch of distributed

cc @madsbk @jakirkham if you have any thoughts. I was hoping that #4595 would have fixed this issue too, but I'm able to reproduce with the latest dev version of distributed

[jakirkham]

Yep this is duplicate of issue ( #3716 ). Would propose we close and track there

[bnaul]

@jakirkham just curious, is there any reason this would have recently (since the previous release) become a much more common issue? the linked issue is a year old but I haven't seen any of the errors from this issue until very recently. would be nice if there's a workaround for now to unblock upgrading to 2021.3

[mrocklin]

It looks like we're expecting a list here when getting a tuple. I suspect that this is due to msgpack always returning tuples for us. We should maybe not mutate this list, but instead make a new one?

[mrocklin]

This maybe solve things? #4653