distributed.scheduler - INFO - Remove worker distributed.core - INFO - Removing comms to tcp://127.0.0.1:39155 distributed.scheduler - DEBUG - Removed worker distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33240': {'op': 'heartbeat_worker', 'address': 'tcp://127.0.0.1:36017', 'now': 1587230034.4553661, 'metrics': {'cpu': 170.3, 'memory': 1303924736, 'time': 1587230033.9736848, 'read_bytes': 60262.5998623573, 'write_bytes': 60262.5998623573, 'num_fds': 28, 'executing': 2, 'in_memory': 49, 'ready': 31, 'in_flight': 0, 'bandwidth': {'total': 330941914.71925104, 'workers': {}, 'types': {}}}, 'reply': True} distributed.core - DEBUG - Calling into handler heartbeat_worker distributed.worker - DEBUG - Ensure communicating. Pending: 1. Connections: 0/50 distributed.worker - DEBUG - Deleted 1 keys distributed.worker - DEBUG - Deleted 1 keys distributed.worker - DEBUG - Deleted 1 keys distributed.worker - DEBUG - Deleted 1 keys distributed.worker - DEBUG - Request 1 keys distributed.worker - DEBUG - Ensure communicating. Pending: 1. Connections: 0/50 distributed.worker - DEBUG - Deleted 1 keys distributed.worker - DEBUG - Deleted 1 keys distributed.worker - DEBUG - Request 2 keys distributed.core - DEBUG - Message from 'tcp://127.0.0.1:57322': {'op': 'get_data', 'keys': ('stackingestimator-1092f6d495e100a052a3612d7ee38219',), 'who': 'tcp://127.0.0.1:38201', 'max_connections': None, 'reply': True} distributed.worker - DEBUG - Ensure communicating. Pending: 1. Connections: 0/50 distributed.worker - DEBUG - Request 2 keys distributed.core - DEBUG - Calling into handler get_data distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33248': {'op': 'unregister', 'address': 'tcp://127.0.0.1:39155', 'reply': True} distributed.core - DEBUG - Calling into handler remove_worker distributed.client - DEBUG - Client receives message {'op': 'lost-data', 'key': 'nanmean-ca08392d-436d-4287-a6db-599666673fa8'} distributed.client - DEBUG - Client receives message {'op': 'lost-data', 'key': 'nanmean-91d81746-f904-49a7-a11f-b27dd370116e'} distributed.core - DEBUG - Message from 'tcp://127.0.0.1:42352': {'op': 'get_data', 'keys': ('cv-split-db0a18c311e748fde196fcb030bd3e18', 'nystroem-db0a18c311e748fde196fcb030bd3e18'), 'who': 'tcp://127.0.0.1:37027', 'max_connections': None, 'reply': True} distributed.core - DEBUG - Calling into handler get_data => The client seems to be getting the error here distributed.client - DEBUG - Client receives message {'op': 'task-erred', 'key': 'nanmean-3e713753-4b5f-417c-a536-3df486aca211', 'exception': b"\x80\x04\x95\xe9\t\x00\x00\x00\x00\x00\x00\x8c\x15distributed.scheduler\x94\x8c\x0cKilledWorker\x94\x93\x94\x8cK('polynomialfeatures-fit-transform-99ed4ed062bb97727e0b280d84fa74c5', 0, 4)\x94h\x00\x8c\x0bWorkerState\x94\x93\x94)\x81\x94N}\x94(\x8c\x06actors\x94\x8f\x94\x8c\x07address\x94\x8c\x15tcp://127.0.0.1:39155\x94\x8c\tbandwidth\x94J\x00\xe1\xf5\x05\x8c\x05extra\x94}\x94\x8c\x08has_what\x94\x8f\x94\x8c\tlast_seen\x94K\x00\x8c\x0flocal_directory\x94\x8c//proteinsHome/dask-worker-space/worker-d45_anj2\x94\x8c\x0cmemory_limit\x94\x8a\x05\x00\xccG\xf5\x01\x8c\x07metrics\x94}\x94\x8c\x04name\x94K\x03\x8c\x05nanny\x94\x8c\x15tcp://127.0.0.1:33049\x94\x8c\x06nbytes\x94K\x00\x8c\x08nthreads\x94K\x02\x8c\toccupancy\x94K\x00\x8c\x03pid\x94MQ\x12\x8c\nprocessing\x94\x8f\x94(\x8c)cv-split-ad6ef8fe7d52a0dc4951f6e90dbd99af\x94\x8cJ('stackingestimator-fit-transform-bfc1ab1a5b6e8a899759b5cd96579002', 0, 4)\x94\x8c?('selectpercentile-fit-a13472f9d53a3fe11016998a12a3315e', 0, 0)\x94\x8cI('selectpercentile-fit-transform-a13472f9d53a3fe11016998a12a3315e', 0, 1)\x94\x8c)cv-split-db59467a1f19eeb62b1ebfe1eaa526a7\x94\x8cK('polynomialfeatures-fit-transform-99ed4ed062bb97727e0b280d84fa74c5', 0, 1)\x94\x8c3('pipeline-1092f6d495e100a052a3612d7ee38219', 0, 2)\x94\x8cJ('stackingestimator-fit-transform-bfc1ab1a5b6e8a899759b5cd96579002', 0, 0)\x94\x8c@('stackingestimator-fit-eb52abc040aaee779e72a7d54d6e9b0e', 0, 3)\x94\x8cA('nystroem-fit-transform-db0a18c311e748fde196fcb030bd3e18', 0, 0)\x94\x8cI('selectpercentile-fit-transform-a13472f9d53a3fe11016998a12a3315e', 0, 2)\x94\x8c3('pipeline-99fb9371a3292130e440ea8db1dd190e', 0, 1)\x94\x8cE('selectpercentile-transform-a13472f9d53a3fe11016998a12a3315e', 0, 0)\x94\x8cJ('stackingestimator-fit-transform-bfc1ab1a5b6e8a899759b5cd96579002', 0, 2)\x94\x8cI('selectpercentile-fit-transform-a13472f9d53a3fe11016998a12a3315e', 0, 4)\x94\x8c@('stackingestimator-fit-eb52abc040aaee779e72a7d54d6e9b0e', 0, 4)\x94\x8cB('kneighborsregressor-fit-71b996da20350c07275a23573d1ae07f', 0, 4)\x94\x8cF('onehotencoder-fit-transform-99fb9371a3292130e440ea8db1dd190e', 0, 2)\x94\x8c-cv-n-samples-a13472f9d53a3fe11016998a12a3315e\x94\x8c+cv-results-20500e6cf8494a256279420d2c353f2a\x94h\x03\x8c6('fastica-fit-d3b00d564f1fdaf8202b1131bc74f7b9', 0, 1)\x94\x8cF('onehotencoder-fit-transform-99fb9371a3292130e440ea8db1dd190e', 0, 0)\x94\x8cI('selectpercentile-fit-transform-a13472f9d53a3fe11016998a12a3315e', 0, 3)\x94\x8c3('pipeline-8d5325b45abfb9623d978dfaaa77b32f', 0, 2)\x94\x8c)cv-split-b0ff559bc33e5db32a3998dbf72457e5\x94\x8cF('stackingestimator-transform-eb52abc040aaee779e72a7d54d6e9b0e', 0, 3)\x94\x8cJ('stackingestimator-fit-transform-1092f6d495e100a052a3612d7ee38219', 0, 3)\x94\x8c<('fastica-transform-d3b00d564f1fdaf8202b1131bc74f7b9', 0, 1)\x94\x8cF('stackingestimator-transform-eb52abc040aaee779e72a7d54d6e9b0e', 0, 4)\x94\x8c3('pipeline-eb52abc040aaee779e72a7d54d6e9b0e', 0, 2)\x94\x90\x8c\tresources\x94}\x94\x8c\x08services\x94}\x94\x8c\x06status\x94\x8c\x07running\x94\x8c\ntime_delay\x94K\x00\x8c\x0eused_resources\x94}\x94\x8c\x08versions\x94}\x94u\x86\x94b\x86\x94R\x94}\x94(\x8c\x04task\x94h\x03\x8c\x0blast_worker\x94h\x06ub.", 'traceback': None} distributed.core - DEBUG - Message from 'tcp://127.0.0.1:42358': {'op': 'get_data', 'keys': ('stackingestimator-bfc1ab1a5b6e8a899759b5cd96579002', 'cv-split-bfc1ab1a5b6e8a899759b5cd96579002'), 'who': 'tcp://127.0.0.1:36017', 'max_connections': None, 'reply': True} distributed.core - DEBUG - Calling into handler get_data distributed.nanny - WARNING - Restarting worker distributed.process - DEBUG - [] got message {'op': 'start', 'future': ()]>} distributed.client - DEBUG - Release key nanmean-f281287e-2cc5-40d2-87cb-20efecefe2d1 => The Exception traceback starts here Traceback (most recent call last): File "/opt/conda/envs/teapot/lib/python3.7/site-packages/tpot/base.py", line 711, in fit distributed.client - DEBUG - Release key nanmean-058b7099-e118-4696-a7f0-e722373b29a6 distributed.client - DEBUG - Release key nanmean-2bbe9d37-5df6-4378-993d-b86dca0c2497 distributed.client - DEBUG - Release key nanmean-5db3113b-fb62-4822-a147-5b420feb6040 distributed.client - DEBUG - Release key nanmean-f7fb71d6-bcd4-46aa-82d6-7ec6e658a01d distributed.client - DEBUG - Release key nanmean-f2e20f52-7de4-4a09-ae3e-9104af8c0562 distributed.client - DEBUG - Release key nanmean-eda4cf86-2044-46bc-8e95-859642bdf703 distributed.client - DEBUG - Release key nanmean-ef48b601-e06d-4356-9fe5-a0aaaa49075b distributed.client - DEBUG - Release key nanmean-b8b89868-ba77-4fb0-9c7f-4c73bb08666d distributed.client - DEBUG - Release key nanmean-2a533880-90bd-4bac-8fdb-f2e8b81310f0 distributed.client - DEBUG - Release key nanmean-b3eafa70-99df-4603-aac7-7270a33ad566 distributed.client - DEBUG - Release key nanmean-d6ff1be6-8948-4a2e-b7d9-560c1c4b45e8 distributed.client - DEBUG - Release key nanmean-0e47dcae-0622-42fe-b6d2-1fd40bb02a94 distributed.client - DEBUG - Release key nanmean-d6b1e231-c8b3-49cd-a17b-eab5c3d7f020 distributed.client - DEBUG - Release key nanmean-17cf7cdf-ad56-42ab-91ee-1a23426fc75f per_generation_function=self._check_periodic_pipeline File "/opt/conda/envs/teapot/lib/python3.7/site-packages/tpot/gp_deap.py", line 227, in eaMuPlusLambda population[:] = toolbox.evaluate(population) File "/opt/conda/envs/teapot/lib/python3.7/site-packages/tpot/base.py", line 1314, in _evaluate_individuals tmp_result_scores = list(dask.compute(*tmp_result_scores)) File "/opt/conda/envs/teapot/lib/python3.7/site-packages/dask/base.py", line 437, in compute results = schedule(dsk, keys, **kwargs) File "/opt/conda/envs/teapot/lib/python3.7/site-packages/distributed/client.py", line 2595, in get results = self.gather(packed, asynchronous=asynchronous, direct=direct) File "/opt/conda/envs/teapot/lib/python3.7/site-packages/distributed/client.py", line 1893, in gather asynchronous=asynchronous, File "/opt/conda/envs/teapot/lib/python3.7/site-packages/distributed/client.py", line 780, in sync distributed.client - DEBUG - Release key nanmean-39a51900-de67-4b50-a451-5ca5271ed5c4 distributed.client - DEBUG - Release key nanmean-a5595429-244f-44c9-a994-bbdf719c8b38 distributed.client - DEBUG - Release key nanmean-f50fb2b1-cfb1-4fd3-b49a-30b665dc2c54 distributed.client - DEBUG - Release key nanmean-70ed1ff6-f77c-4936-b526-6a342696c918 distributed.client - DEBUG - Release key nanmean-1cd8c10b-8c3c-4180-9575-b03a02744d32 distributed.client - DEBUG - Release key nanmean-84f6d95a-e93d-4c74-91a7-608cec9c8fd3 distributed.client - DEBUG - Release key nanmean-c8f5d425-ea38-4f4d-8e15-6aaea051c392 distributed.client - DEBUG - Release key nanmean-f1fa8328-af56-4fdc-839c-28224c73298d distributed.client - DEBUG - Release key nanmean-67f5bdb1-7316-4ea4-9e54-e1c27a91fad1 distributed.client - DEBUG - Release key nanmean-2294a6dc-497f-4a68-99cf-421fdf91d1c0 distributed.client - DEBUG - Release key nanmean-0e20bf2e-fb76-409a-8cea-ab6c71da5904 distributed.client - DEBUG - Release key nanmean-17533b3f-5262-472f-9ac0-55d1afc198e8 distributed.client - DEBUG - Release key nanmean-d558383a-3edd-47de-bbb2-72f761e05261 distributed.client - DEBUG - Release key nanmean-89482cf5-7a23-4094-9670-4a2f6416fada distributed.client - DEBUG - Release key nanmean-10ac980d-084b-432b-8635-922413b35d64 distributed.client - DEBUG - Release key nanmean-d5da035d-8ce6-4894-89c4-42b8d8cc37a2 distributed.client - DEBUG - Release key nanmean-8fef05bd-9b41-4234-bc1b-7f383873d277 distributed.client - DEBUG - Release key nanmean-a559c411-b32d-40a8-ab21-6064fff46845 distributed.client - DEBUG - Release key nanmean-df6e51f4-8f77-48cc-a00b-c9c708d3e639 distributed.client - DEBUG - Release key nanmean-91d81746-f904-49a7-a11f-b27dd370116e distributed.client - DEBUG - Release key nanmean-f473d64f-c4a3-44e0-ae1e-646d959a57f3 distributed.client - DEBUG - Release key nanmean-fc0138e7-cad5-4b3b-97c1-8528d32c757c distributed.client - DEBUG - Release key nanmean-43ad0943-1645-419b-bbb2-0a4e6be46e68 distributed.client - DEBUG - Release key nanmean-42ea5435-4784-4b41-a537-56d94a091aba distributed.client - DEBUG - Release key nanmean-c30e2586-3101-4f78-9341-ca45f56ed4ec distributed.client - DEBUG - Release key nanmean-729db0d9-9bf2-46d7-9a20-11493df8e9be distributed.client - DEBUG - Release key nanmean-8e75924f-6e03-4bcb-b326-a1d336650aad distributed.client - DEBUG - Release key nanmean-77e89ddb-74bf-42bd-9dc9-b011066ea563 distributed.client - DEBUG - Release key nanmean-78ecd0a1-3fea-4be9-832c-d37ca6f79064 distributed.client - DEBUG - Release key nanmean-3e713753-4b5f-417c-a536-3df486aca211 distributed.client - DEBUG - Release key nanmean-39b3e20a-546d-4faa-96f6-3b3b5dfe592f distributed.client - DEBUG - Release key nanmean-c13d9a09-35d8-42a1-a531-d1559e6a2237 distributed.client - DEBUG - Release key nanmean-fe68e829-109e-4d78-9e38-da56b3a6e995 distributed.client - DEBUG - Release key nanmean-60c1c4a1-cf96-48f5-9fc8-1faba88b8c63 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs File "/opt/conda/envs/teapot/lib/python3.7/site-packages/distributed/utils.py", line 348, in sync raise exc.with_traceback(tb) File "/opt/conda/envs/teapot/lib/python3.7/site-packages/distributed/utils.py", line 332, in f result[0] = yield future File "/opt/conda/envs/teapot/lib/python3.7/site-packages/tornado/gen.py", line 735, in run value = future.result() File "/opt/conda/envs/teapot/lib/python3.7/site-packages/distributed/client.py", line 1752, in _gather raise exception.with_traceback(traceback) distributed.scheduler.KilledWorker: ("('polynomialfeatures-fit-transform-99ed4ed062bb97727e0b280d84fa74c5', 0, 4)", ) => I wonder if that was called prematurely? During handling of the above exception, another exception occurred: Traceback (most recent call last): File "train_test_tpot_model.py", line 76, in ===> This is my code main() File "train_test_tpot_model.py", line 53, in main regressor = regressor_finder.fit(tpot_train_X, tpot_train_Y, groups=tpot_train_G) File "/opt/conda/envs/teapot/lib/python3.7/site-packages/tpot/base.py", line 742, in fit raise e File "/opt/conda/envs/teapot/lib/python3.7/site-packages/tpot/base.py", line 733, in fit self._update_top_pipeline() File "/opt/conda/envs/teapot/lib/python3.7/site-packages/tpot/base.py", line 811, in _update_top_pipeline raise RuntimeError('A pipeline has not yet been optimized. Please call fit() first.') RuntimeError: A pipeline has not yet been optimized. Please call fit() first. distributed.process - DEBUG - [] created process with pid 4717 distributed.client - DEBUG - Release key nanmean-2483bbf8-190b-4f71-b511-cb2cdfc0a831 distributed.process - INFO - reaping stray process distributed.client - DEBUG - Release key nanmean-f935ba91-ce60-4e2b-893a-d32ac8cf14af distributed.process - INFO - reaping stray process distributed.process - INFO - reaping stray process distributed.process - INFO - reaping stray process distributed.client - DEBUG - Release key nanmean-ca08392d-436d-4287-a6db-599666673fa8 distributed.process - DEBUG - [] process 4717 exited with code -15 distributed.client - DEBUG - Release key nanmean-4fbc0454-f235-47e5-a6c7-ef2d2fe9ef96 distributed.client - DEBUG - Release key nanmean-958dcb9b-4d83-4772-a758-a224ba949a48 distributed.client - DEBUG - Release key nanmean-87b7b487-6074-40d2-bef6-066c6dbb94b7 distributed.client - DEBUG - Release key nanmean-c4706647-82d4-445b-8bd9-1a89928f7da3 distributed.client - DEBUG - Release key nanmean-b3edaeb0-b6dc-4493-9a9b-9b1ea26db23a distributed.client - DEBUG - Release key nanmean-c378af96-bbca-40c1-824a-add675f1e9f3 distributed.client - DEBUG - Release key nanmean-7f938e7d-7272-4e41-962e-f51a056314fe distributed.client - DEBUG - Release key nanmean-bde96b23-beb0-41c7-a722-e3c9d4c2430e distributed.nanny - INFO - Worker process 4717 was killed by signal 15 distributed.process - DEBUG - [] got message {'op': 'stop'} distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-f281287e-2cc5-40d2-87cb-20efecefe2d1',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-058b7099-e118-4696-a7f0-e722373b29a6',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-2bbe9d37-5df6-4378-993d-b86dca0c2497',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-5db3113b-fb62-4822-a147-5b420feb6040',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-f7fb71d6-bcd4-46aa-82d6-7ec6e658a01d',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-f2e20f52-7de4-4a09-ae3e-9104af8c0562',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-eda4cf86-2044-46bc-8e95-859642bdf703',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-ef48b601-e06d-4356-9fe5-a0aaaa49075b',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-b8b89868-ba77-4fb0-9c7f-4c73bb08666d',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-2a533880-90bd-4bac-8fdb-f2e8b81310f0',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-b3eafa70-99df-4603-aac7-7270a33ad566',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-d6ff1be6-8948-4a2e-b7d9-560c1c4b45e8',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-0e47dcae-0622-42fe-b6d2-1fd40bb02a94',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-d6b1e231-c8b3-49cd-a17b-eab5c3d7f020',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-17cf7cdf-ad56-42ab-91ee-1a23426fc75f',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-39a51900-de67-4b50-a451-5ca5271ed5c4',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-a5595429-244f-44c9-a994-bbdf719c8b38',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-f50fb2b1-cfb1-4fd3-b49a-30b665dc2c54',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-70ed1ff6-f77c-4936-b526-6a342696c918',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-1cd8c10b-8c3c-4180-9575-b03a02744d32',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-84f6d95a-e93d-4c74-91a7-608cec9c8fd3',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-c8f5d425-ea38-4f4d-8e15-6aaea051c392',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-f1fa8328-af56-4fdc-839c-28224c73298d',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-67f5bdb1-7316-4ea4-9e54-e1c27a91fad1',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-2294a6dc-497f-4a68-99cf-421fdf91d1c0',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-0e20bf2e-fb76-409a-8cea-ab6c71da5904',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-17533b3f-5262-472f-9ac0-55d1afc198e8',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-d558383a-3edd-47de-bbb2-72f761e05261',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-89482cf5-7a23-4094-9670-4a2f6416fada',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-10ac980d-084b-432b-8635-922413b35d64',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-d5da035d-8ce6-4894-89c4-42b8d8cc37a2',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-8fef05bd-9b41-4234-bc1b-7f383873d277',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-a559c411-b32d-40a8-ab21-6064fff46845',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-df6e51f4-8f77-48cc-a00b-c9c708d3e639',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-91d81746-f904-49a7-a11f-b27dd370116e',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-f473d64f-c4a3-44e0-ae1e-646d959a57f3',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-fc0138e7-cad5-4b3b-97c1-8528d32c757c',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-43ad0943-1645-419b-bbb2-0a4e6be46e68',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-42ea5435-4784-4b41-a537-56d94a091aba',) distributed.process - DEBUG - [] process 4636 exited with code -15 distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-c30e2586-3101-4f78-9341-ca45f56ed4ec',) distributed.process - DEBUG - [] process 4664 exited with code -15 distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-729db0d9-9bf2-46d7-9a20-11493df8e9be',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-8e75924f-6e03-4bcb-b326-a1d336650aad',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-77e89ddb-74bf-42bd-9dc9-b011066ea563',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-78ecd0a1-3fea-4be9-832c-d37ca6f79064',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-3e713753-4b5f-417c-a536-3df486aca211',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-39b3e20a-546d-4faa-96f6-3b3b5dfe592f',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-c13d9a09-35d8-42a1-a531-d1559e6a2237',) distributed.process - DEBUG - [] process 4426 exited with code -15 distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-fe68e829-109e-4d78-9e38-da56b3a6e995',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-60c1c4a1-cf96-48f5-9fc8-1faba88b8c63',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-2483bbf8-190b-4f71-b511-cb2cdfc0a831',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-f935ba91-ce60-4e2b-893a-d32ac8cf14af',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-ca08392d-436d-4287-a6db-599666673fa8',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-4fbc0454-f235-47e5-a6c7-ef2d2fe9ef96',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-958dcb9b-4d83-4772-a758-a224ba949a48',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-87b7b487-6074-40d2-bef6-066c6dbb94b7',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-c4706647-82d4-445b-8bd9-1a89928f7da3',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-b3edaeb0-b6dc-4493-9a9b-9b1ea26db23a',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-c378af96-bbca-40c1-824a-add675f1e9f3',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-7f938e7d-7272-4e41-962e-f51a056314fe',) distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: ('nanmean-bde96b23-beb0-41c7-a722-e3c9d4c2430e',) distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33248': {'op': 'unregister', 'address': 'tcp://127.0.0.1:39155', 'reply': True} distributed.core - DEBUG - Calling into handler remove_worker distributed.nanny - INFO - Worker process 4636 was killed by signal 15 distributed.process - DEBUG - [] got message {'op': 'stop'} distributed.nanny - INFO - Worker process 4664 was killed by signal 15 distributed.process - DEBUG - [] got message {'op': 'stop'} distributed.nanny - INFO - Worker process 4426 was killed by signal 15 distributed.process - DEBUG - [] got message {'op': 'stop'} distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33240' while reading message: in : Stream is closed. Last operation: heartbeat_worker distributed.scheduler - INFO - Remove worker distributed.core - INFO - Removing comms to tcp://127.0.0.1:36017 distributed.scheduler - DEBUG - Removed worker distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33222' while reading message: in : Stream is closed. Last operation: heartbeat_worker distributed.scheduler - INFO - Remove worker distributed.core - INFO - Removing comms to tcp://127.0.0.1:38201 distributed.scheduler - DEBUG - Removed worker distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33146' while reading message: in : Stream is closed. Last operation: heartbeat_worker distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33082' while reading message: in : Stream is closed. Last operation: identity distributed.scheduler - INFO - Remove client Client-worker-1ffa32f8-8194-11ea-914a-dd1e73e32ba1 distributed.scheduler - DEBUG - Client Client-worker-1ffa32f8-8194-11ea-914a-dd1e73e32ba1 releases keys: [] distributed.scheduler - DEBUG - Finished handling client Client-worker-1ffa32f8-8194-11ea-914a-dd1e73e32ba1 distributed.scheduler - INFO - Remove worker distributed.core - INFO - Removing comms to tcp://127.0.0.1:37027 distributed.scheduler - INFO - Lost all workers distributed.scheduler - DEBUG - Removed worker distributed.nanny - WARNING - Restarting worker distributed.process - DEBUG - [] got message {'op': 'start', 'future': ()]>} distributed.batched - INFO - Batched Comm Closed: in : Stream is closed distributed.batched - INFO - Batched Comm Closed: in : Stream is closed distributed.batched - INFO - Batched Comm Closed: in : Stream is closed distributed.process - DEBUG - [] created process with pid 4720 distributed.comm.tcp - DEBUG - Incoming connection from 'tcp://127.0.0.1:33442' to 'tcp://127.0.0.1:44991' distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2 distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms distributed.core - DEBUG - Connection from 'tcp://127.0.0.1:33442' to Scheduler distributed.scheduler - INFO - Close client connection: Client-worker-1ffa32f8-8194-11ea-914a-dd1e73e32ba1 distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2 distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33212': {'op': 'unregister', 'address': 'tcp://127.0.0.1:38201', 'reply': True} distributed.core - DEBUG - Calling into handler remove_worker distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33230': {'op': 'unregister', 'address': 'tcp://127.0.0.1:36017', 'reply': True} distributed.core - DEBUG - Calling into handler remove_worker distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Restarting worker distributed.process - DEBUG - [] got message {'op': 'start', 'future': ()]>} distributed.process - DEBUG - [] created process with pid 4723 distributed.comm.tcp - DEBUG - Incoming connection from 'tcp://127.0.0.1:33444' to 'tcp://127.0.0.1:44991' distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2 distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms distributed.process - DEBUG - [] got message {'op': 'start', 'future': ()]>} distributed.core - DEBUG - Connection from 'tcp://127.0.0.1:33444' to Scheduler distributed.process - DEBUG - [] created process with pid 4726 distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2 distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33442': {'op': 'retire_workers', 'workers': (0, 1, 2, 3), 'reply': True} distributed.core - DEBUG - Calling into handler retire_workers distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33444': {'op': 'unregister', 'address': 'tcp://127.0.0.1:37027', 'reply': True} distributed.core - DEBUG - Calling into handler remove_worker distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:39257' distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:33049' distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:45447' distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:33353' distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33444' while reading message: in : Stream is closed. Last operation: unregister distributed.diskutils - DEBUG - Checking lock file '/proteinsHome/dask-worker-space/worker-d45_anj2.dirlock'... distributed.diskutils - DEBUG - Checking lock file '/proteinsHome/dask-worker-space/worker-le3aipb5.dirlock'... distributed.diskutils - DEBUG - Checking lock file '/proteinsHome/dask-worker-space/worker-17aintv4.dirlock'... distributed.diskutils - DEBUG - Checking lock file '/proteinsHome/dask-worker-space/worker-cmf1m0mj.dirlock'... distributed.diskutils - DEBUG - Locking '/proteinsHome/dask-worker-space/worker-a2dovcmt.dirlock'... distributed.worker - INFO - Start worker at: tcp://127.0.0.1:39771 distributed.worker - INFO - Listening to: tcp://127.0.0.1:39771 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44991 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Threads: 2 distributed.worker - INFO - Memory: 8.41 GB distributed.worker - INFO - Local Directory: /proteinsHome/dask-worker-space/worker-a2dovcmt distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:39771 distributed.worker - INFO - Closed worker has not yet started: None distributed.diskutils - DEBUG - Locking '/proteinsHome/dask-worker-space/worker-0wnalw6y.dirlock'... distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33363 distributed.worker - INFO - Listening to: tcp://127.0.0.1:33363 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44991 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Threads: 2 distributed.worker - INFO - Memory: 8.41 GB distributed.worker - INFO - Local Directory: /proteinsHome/dask-worker-space/worker-0wnalw6y distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33363 distributed.worker - INFO - Closed worker has not yet started: None distributed.diskutils - DEBUG - Locking '/proteinsHome/dask-worker-space/worker-ej7_qh38.dirlock'... distributed.worker - INFO - Start worker at: tcp://127.0.0.1:41755 distributed.worker - INFO - Listening to: tcp://127.0.0.1:41755 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44991 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Threads: 2 distributed.worker - INFO - Memory: 8.41 GB distributed.worker - INFO - Local Directory: /proteinsHome/dask-worker-space/worker-ej7_qh38 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:41755 distributed.worker - INFO - Closed worker has not yet started: None distributed.process - DEBUG - [] process 4720 exited with code 0 distributed.process - DEBUG - [] got message {'op': 'stop'} distributed.process - DEBUG - [] process 4723 exited with code 0 distributed.process - DEBUG - [] got message {'op': 'stop'} distributed.process - DEBUG - [] process 4726 exited with code 0 distributed.process - DEBUG - [] got message {'op': 'stop'} distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33212' while reading message: in : Stream is closed. Last operation: unregister distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33248' while reading message: in : Stream is closed. Last operation: unregister distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33230' while reading message: in : Stream is closed. Last operation: unregister distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33442': {'op': 'close', 'close_workers': True, 'reply': True} distributed.scheduler - INFO - Scheduler closing... distributed.scheduler - INFO - Scheduler closing all comms distributed.scheduler - DEBUG - Client Client-1f9d051e-8194-11ea-913d-a1d14a018719 releases keys: [] distributed.scheduler - DEBUG - Finished handling client Client-1f9d051e-8194-11ea-913d-a1d14a018719 distributed.client - INFO - Client report stream closed to scheduler distributed.client - INFO - Reconnecting... distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.core - DEBUG - Message from 'tcp://127.0.0.1:33074': {'op': 'identity', 'reply': True} distributed.core - DEBUG - Calling into handler identity distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33054' while reading message: in : Stream is closed. Last operation: subscribe_worker_status distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33066' while reading message: in : Stream is closed. Last operation: identity distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33074' while reading message: in : Stream is closed. Last operation: identity distributed.core - DEBUG - Lost connection to 'tcp://127.0.0.1:33112' while reading message: in : Stream is closed. Last operation: identity distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect distributed.comm.core - DEBUG - sleeping on connect => Thus is the end of the log