TypeError in distributed.comm.tcp.convert_stream_closed_error

[hendrikmakait]

Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: Task exception was never retrieved
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: future: <Task finished name='Task-103' coro=<Server._handle_comm() done, defined at /opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/core.py:675> exception=TypeError("argument of type 'NoneType' is not iterable")>
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: Traceback (most recent call last):
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 223, in read
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:     frames_nbytes = await stream.read_bytes(fmt_size)
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: tornado.iostream.StreamClosedError: Stream is closed
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: During handling of the above exception, another exception occurred:
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: Traceback (most recent call last):
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/core.py", line 769, in _handle_comm
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:     result = await result
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/worker.py", line 1692, in get_data
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:     response = await comm.read(deserializers=serializers)
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 239, in read
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:     convert_stream_closed_error(self, e)
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 140, in convert_stream_closed_error
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]:     if "UNKNOWN_CA" in exc.reason:
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: TypeError: argument of type 'NoneType' is not iterable

(https://cloud.coiled.io/dask-engineering/clusters/48659/details on hendrik-debug-worker-e73a905446)

Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: 2022-08-11 14:00:05,617 - distributed.core - ERROR - argument of type 'NoneType' is not iterable
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: Traceback (most recent call last):
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 223, in read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     frames_nbytes = await stream.read_bytes(fmt_size)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 426, in read_bytes
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     self._try_inline_read()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 841, in _try_inline_read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     self._check_closed()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 1017, in _check_closed
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     raise StreamClosedError(real_error=self.error)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: tornado.iostream.StreamClosedError: Stream is closed
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: During handling of the above exception, another exception occurred:
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: Traceback (most recent call last):
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/core.py", line 821, in handle_stream
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     msgs = await comm.read()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 239, in read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     convert_stream_closed_error(self, e)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 140, in convert_stream_closed_error
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     if "UNKNOWN_CA" in exc.reason:
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: TypeError: argument of type 'NoneType' is not iterable
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: 2022-08-11 14:00:05,627 - distributed.worker - ERROR - argument of type 'NoneType' is not iterable
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: Traceback (most recent call last):
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 223, in read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     frames_nbytes = await stream.read_bytes(fmt_size)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 426, in read_bytes
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     self._try_inline_read()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 841, in _try_inline_read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     self._check_closed()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 1017, in _check_closed
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     raise StreamClosedError(real_error=self.error)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: tornado.iostream.StreamClosedError: Stream is closed
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: During handling of the above exception, another exception occurred:
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: Traceback (most recent call last):
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/worker.py", line 180, in wrapper
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     return await method(self, *args, **kwargs)  # type: ignore
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/worker.py", line 1211, in handle_scheduler
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     await self.handle_stream(comm)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/core.py", line 821, in handle_stream
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     msgs = await comm.read()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 239, in read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     convert_stream_closed_error(self, e)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:   File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 140, in convert_stream_closed_error
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]:     if "UNKNOWN_CA" in exc.reason:
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: TypeError: argument of type 'NoneType' is not iterable

(https://cloud.coiled.io/dask-engineering/clusters/48659/details on hendrik-debug-worker-8ede19223f)