/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/geoapis/vector.py:270: ShapelyDeprecationWarning: The 'GeometryType()' method is deprecated, and will be removed in the future. You can use the 'geom_type' attribute instead.
  shapely_geometry.geometryType() == "MultiPolygon"

0GB [00:00, ?GB/s]
0GB [00:00, ?GB/s]
/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 36138 instead
  warnings.warn(
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
2023-10-16 10:05:52,766 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:32814
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:32814 after 30 s
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
2023-10-16 10:07:28,261 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:43998
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:43998 after 30 s
2023-10-16 10:07:32,785 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:38969
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:38969 after 30 s
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
2023-10-16 10:08:36,042 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:41271
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:41271 after 30 s
2023-10-16 10:08:30,189 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:41756
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:41756 after 30 s
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
2023-10-16 10:09:50,263 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:41453
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:41453 after 30 s
2023-10-16 10:09:50,266 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:45311
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:45311 after 30 s
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
2023-10-16 10:11:27,622 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:43279
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:43279 after 30 s
2023-10-16 10:11:27,902 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:39047
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:39047 after 30 s
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
2023-10-16 10:11:28,413 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:40615
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:40615 after 30 s
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
2023-10-16 10:11:54,399 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:33384
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:33384 after 30 s
2023-10-16 10:11:54,401 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:33523
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:33523 after 30 s
2023-10-16 10:12:09,952 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:42963
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect
    stream = await self.client.connect(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect
    af, addr, stream = await connector.start(connect_timeout=timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect
    comm = await wait_for(
           ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep
    response = await get_data_from_worker(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker
    comm = await rpc.connect(worker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect
    return connect_attempt.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect
    comm = await connect(
           ^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect
    raise OSError(
OSError: Timed out trying to connect to tcp://127.0.0.1:42963 after 30 s
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
WARNING:root:In dem.roughness_over_chunk the latest chunk has no data and is being ignored.
2023-10-16 10:22:58,606 - distributed.nanny - WARNING - Worker process still alive after 3.1999987792968754 seconds, killing
2023-10-16 10:22:58,621 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-10-16 10:22:58,621 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,622 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,622 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,622 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-10-16 10:22:58,623 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing
2023-10-16 10:22:58,623 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-10-16 10:22:58,623 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,624 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,624 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,624 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing
2023-10-16 10:22:58,624 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,624 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,625 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,625 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,625 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,625 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing
2023-10-16 10:22:58,625 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,626 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,626 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,626 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-10-16 10:22:58,627 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,627 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,627 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-10-16 10:22:58,627 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,627 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,628 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,629 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-10-16 10:22:58,629 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,629 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,630 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-10-16 10:22:58,630 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,631 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,632 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,632 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,635 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,635 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing
2023-10-16 10:22:58,635 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-10-16 10:22:58,636 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing
2023-10-16 10:22:59,415 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x2aab6c9eb2d0>>, <Task finished name='Task-2993411' coro=<SpecCluster._correct_state_internal() done, defined at /nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/spec.py:346> exception=TimeoutError()>)
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/ioloop.py", line 738, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/ioloop.py", line 762, in _discard_future_result
    future.result()
TimeoutError
Traceback (most recent call last):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for
    return await fut
           ^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/pearsonra/cylc-run/cylc-geofabrics/run2/lib/python/roughness.py", line 62, in main
    runner.run()
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/geofabrics/processor.py", line 1181, in run
    with cluster, distributed.Client(cluster) as client:
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/cluster.py", line 540, in __exit__
    aw = self.close()
         ^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/spec.py", line 293, in close
    aw = super().close(timeout)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/cluster.py", line 226, in close
    return self.sync(self._close, callback_timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 359, in sync
    return sync(
           ^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 426, in sync
    raise exc.with_traceback(tb)
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 399, in f
    result = yield future
             ^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/gen.py", line 767, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/spec.py", line 446, in _close
    await self._correct_state()
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/spec.py", line 359, in _correct_state_internal
    await asyncio.gather(*tasks)
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/nanny.py", line 595, in close
    await self.kill(timeout=timeout, reason=reason)
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/nanny.py", line 380, in kill
    await self.process.kill(reason=reason, timeout=0.8 * (deadline - time()))
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/nanny.py", line 843, in kill
    await process.join(max(0, deadline - time()))
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/process.py", line 330, in join
    await wait_for(asyncio.shield(self._exit_future), timeout)
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for
    async with asyncio.timeout(timeout):
  File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError
2023-10-16T10:23:14+13:00 CRITICAL - failed/ERR