⚠️ CI failed ⚠️

[github-actions]

Workflow Run URL

[fjetter]

#939

[hendrikmakait]

Looking at the logs, this appears to have been an "ordinary" connection timeout (though I'm not sure what caused it).

__________________________ test_set_index[1-p2p-True] __________________________
[gw2] linux -- Python 3.9.16 /usr/share/miniconda3/envs/test/bin/python

>   ???

/opt/coiled/env/lib/python3.9/site-packages/tornado/iostream.py:869: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/coiled/env/lib/python3.9/site-packages/tornado/iostream.py:1601: in read_from_fd
    ???
/opt/coiled/env/lib/python3.9/ssl.py:1242: in recv_into
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   TimeoutError: [Errno 110] Connection timed out

/opt/coiled/env/lib/python3.9/ssl.py:1100: TimeoutError

The above exception was the direct cause of the following exception:

>   ???

/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_shuffle.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_worker_plugin.py:669: in add_partition
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/utils.py:426: in sync
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/utils.py:399: in f
    ???
/opt/coiled/env/lib/python3.9/site-packages/tornado/gen.py:769: in run
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_worker_plugin.py:534: in add_partition
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_worker_plugin.py:151: in _write_to_comm
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_buffer.py:189: in write
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_buffer.py:107: in process
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_comms.py:72: in _process
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_worker_plugin.py:122: in send
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/core.py:1374: in send_recv_from_rpc
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/core.py:1133: in send_recv
    ???
/opt/coiled/env/lib/python3.9/site-packages/distributed/comm/tcp.py:240: in read
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   distributed.comm.core.CommClosedError: in <TLS (closed) ConnectionPool.shuffle_receive local=tls://10.0.17.33:53666 remote=tls://10.0.22.161:41821>: TimeoutError: [Errno 110] Connection timed out

/opt/coiled/env/lib/python3.9/site-packages/distributed/comm/tcp.py:141: CommClosedError

The above exception was the direct cause of the following exception:

small_client = <Client: 'tls://10.0.25.134:8786' processes=10 threads=20, memory=71.37 GiB>
persist = True, memory_multiplier = 1, configure_shuffling = None

    @pytest.mark.parametrize("persist", [True, False])
    def test_set_index(small_client, persist, memory_multiplier, configure_shuffling):
        memory = cluster_memory(small_client)  # 76.66 GiB
    
        df_big = timeseries_of_size(
            memory * memory_multiplier, dtypes={str(i): float for i in range(100)}
        )  # 66.58 MiB partitions
        df_big["predicate"] = df_big["0"] * 1e9
        df_big = df_big.astype({"predicate": "int"})
        if persist:
            df_big = df_big.persist()
        df_indexed = df_big.set_index("0")
>       wait(df_indexed.size, small_client, 20 * 60)

tests/benchmarks/test_join.py:73: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/utils_test.py:176: in wait
    raise f.exception()
/opt/coiled/env/lib/python3.9/site-packages/distributed/worker.py:3129: in apply_function_simple
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   RuntimeError: shuffle_transfer failed during shuffle [462](https://github.com/coiled/benchmarks/actions/runs/5796983630/job/15711638127#step:8:463)96374d8f61e13fd5439b19e6751f0

/opt/coiled/env/lib/python3.9/site-packages/distributed/shuffle/_shuffle.py:76: RuntimeError