Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

getting Fatal Python error: Bus error or ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed #875

Closed
yahoyoungho opened this issue Nov 18, 2019 · 2 comments
Labels
Ray ⚡ Issues related to the Ray engine

Comments

@yahoyoungho
Copy link

yahoyoungho commented Nov 18, 2019

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
    Linux 4.19.0-5-amd64 [CI] Adding travis config #1 SMP Debian 4.19.37-2~mx17+1 (2019-05-15) x86_64 GNU/Linux
  • Modin installed from (source or binary):
    pip3 install "modin[complete]"
  • Modin version:
    0.6.3
  • Python version:
    3.5.3
  • Exact command to reproduce:
    import modin.pandas as pd
    mat=pd.read_csv("refOriMatzero.vcf",sep="\s+")
    mat = mat.set_index("ID")
    mat = mat.T
    mat_np = mat.to_numpy()

Describe the problem

after loading the big data (around 20GB) I am trying to do some preprocessing using sklearn packages.
I have tried two functions,

  1. scaled_features = StandardScaler().fit_transform(mat.values)
  2. mat_np = mat.to_numpy()

Source code / logs

(pid=41705) Fatal Python error: Bus error
(pid=41705) 
(pid=41708) Fatal Python error: Bus error
(pid=41708) 
(pid=41646) Traceback (most recent call last):
(pid=41646)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41646)     ray.worker.global_worker.main_loop()
(pid=41646)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41646)     task = self._get_next_task_from_raylet()
(pid=41646)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41646)     task = self.raylet_client.get_task()
(pid=41646)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41646)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41646) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41646) 
(pid=41646) During handling of the above exception, another exception occurred:
(pid=41646) 
(pid=41646) Traceback (most recent call last):
(pid=41646)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41646)     job_id=None)
(pid=41646)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41646)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41646)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41646)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41646) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41651) Traceback (most recent call last):
(pid=41651)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41651)     ray.worker.global_worker.main_loop()
(pid=41651)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41651)     task = self._get_next_task_from_raylet()
(pid=41651)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41651)     task = self.raylet_client.get_task()
(pid=41651)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41651)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41651) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41651) 
(pid=41651) During handling of the above exception, another exception occurred:
(pid=41651) 
(pid=41651) Traceback (most recent call last):
(pid=41651)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41651)     job_id=None)
(pid=41651)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41651)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41651)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41651)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41651) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41712) Traceback (most recent call last):
(pid=41712)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41712)     ray.worker.global_worker.main_loop()
(pid=41712)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41712)     task = self._get_next_task_from_raylet()
(pid=41712)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41712)     task = self.raylet_client.get_task()
(pid=41712)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41712)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41712) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41712) 
(pid=41712) During handling of the above exception, another exception occurred:
(pid=41712) 
(pid=41712) Traceback (most recent call last):
(pid=41712)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41712)     job_id=None)
(pid=41712)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41712)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41712)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41712)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41712) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41657) Traceback (most recent call last):
(pid=41657)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41657)     ray.worker.global_worker.main_loop()
(pid=41657)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41657)     task = self._get_next_task_from_raylet()
(pid=41657)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41657)     task = self.raylet_client.get_task()
(pid=41657)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41657)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41657) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41657) 
(pid=41657) During handling of the above exception, another exception occurred:
(pid=41657) 
(pid=41657) Traceback (most recent call last):
(pid=41657)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41657)     job_id=None)
(pid=41657)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41657)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41657)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41657)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41657) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41660) Traceback (most recent call last):
(pid=41660)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41660)     ray.worker.global_worker.main_loop()
(pid=41660)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41660)     task = self._get_next_task_from_raylet()
(pid=41660)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41660)     task = self.raylet_client.get_task()
(pid=41660)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41660)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41660) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41660) 
(pid=41660) During handling of the above exception, another exception occurred:
(pid=41660) 
(pid=41660) Traceback (most recent call last):
(pid=41660)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41660)     job_id=None)
(pid=41660)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41660)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41660)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41660)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41660) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41650) Traceback (most recent call last):
(pid=41650)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41650)     ray.worker.global_worker.main_loop()
(pid=41650)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41650)     task = self._get_next_task_from_raylet()
(pid=41650)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41650)     task = self.raylet_client.get_task()
(pid=41650)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41650)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41650) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41650) 
(pid=41650) During handling of the above exception, another exception occurred:
(pid=41650) 
(pid=41650) Traceback (most recent call last):
(pid=41650)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41650)     job_id=None)
(pid=41650)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41650)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41650)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41650)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41650) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41656) Traceback (most recent call last):
(pid=41656)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41656)     ray.worker.global_worker.main_loop()
(pid=41656)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41656)     task = self._get_next_task_from_raylet()
(pid=41656)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41656)     task = self.raylet_client.get_task()
(pid=41656)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41656)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41656) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41656) 
(pid=41656) During handling of the above exception, another exception occurred:
(pid=41656) 
(pid=41656) Traceback (most recent call last):
(pid=41656)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41656)     job_id=None)
(pid=41656)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41656)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41656)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41656)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41656) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41666) Traceback (most recent call last):
(pid=41666)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41666)     ray.worker.global_worker.main_loop()
(pid=41666)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41666)     task = self._get_next_task_from_raylet()
(pid=41666)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41666)     task = self.raylet_client.get_task()
(pid=41666)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41666)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41666) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41666) 
(pid=41666) During handling of the above exception, another exception occurred:
(pid=41666) 
(pid=41666) Traceback (most recent call last):
(pid=41666)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41666)     job_id=None)
(pid=41666)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41666)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41666)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41666)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41666) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41654) Traceback (most recent call last):
(pid=41654)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41654)     ray.worker.global_worker.main_loop()
(pid=41654)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41654)     task = self._get_next_task_from_raylet()
(pid=41654)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41654)     task = self.raylet_client.get_task()
(pid=41654)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41654)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41654) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41654) 
(pid=41654) During handling of the above exception, another exception occurred:
(pid=41654) 
(pid=41654) Traceback (most recent call last):
(pid=41654)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41654)     job_id=None)
(pid=41654)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41654)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41654)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41654)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41654) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41672) Traceback (most recent call last):
(pid=41672)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41672)     ray.worker.global_worker.main_loop()
(pid=41672)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41672)     task = self._get_next_task_from_raylet()
(pid=41672)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41672)     task = self.raylet_client.get_task()
(pid=41672)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41672)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41672) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41672) 
(pid=41672) During handling of the above exception, another exception occurred:
(pid=41672) 
(pid=41672) Traceback (most recent call last):
(pid=41672)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41672)     job_id=None)
(pid=41672)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41672)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41672)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41672)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41672) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41723) Traceback (most recent call last):
(pid=41723)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41723)     ray.worker.global_worker.main_loop()
(pid=41723)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41723)     task = self._get_next_task_from_raylet()
(pid=41723)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41723)     task = self.raylet_client.get_task()
(pid=41723)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41723)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41723) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41723) 
(pid=41723) During handling of the above exception, another exception occurred:
(pid=41723) 
(pid=41723) Traceback (most recent call last):
(pid=41723)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41723)     job_id=None)
(pid=41723)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41723)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41723)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41723)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41723) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41658) Traceback (most recent call last):
(pid=41658)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41658)     ray.worker.global_worker.main_loop()
(pid=41658)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41658)     task = self._get_next_task_from_raylet()
(pid=41658)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41658)     task = self.raylet_client.get_task()
(pid=41658)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41658)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41658) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41658) 
(pid=41658) During handling of the above exception, another exception occurred:
(pid=41658) 
(pid=41658) Traceback (most recent call last):
(pid=41658)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41658)     job_id=None)
(pid=41658)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41658)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41658)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41658)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41658) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41703) Traceback (most recent call last):
(pid=41703)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41703)     ray.worker.global_worker.main_loop()
(pid=41703)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41703)     task = self._get_next_task_from_raylet()
(pid=41703)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41703)     task = self.raylet_client.get_task()
(pid=41703)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41703)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41703) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41703) 
(pid=41703) During handling of the above exception, another exception occurred:
(pid=41703) 
(pid=41703) Traceback (most recent call last):
(pid=41703)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41703)     job_id=None)
(pid=41703)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41703)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41703)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41703)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41703) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41700) Traceback (most recent call last):
(pid=41700)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41700)     ray.worker.global_worker.main_loop()
(pid=41700)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41700)     task = self._get_next_task_from_raylet()
(pid=41700)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41700)     task = self.raylet_client.get_task()
(pid=41700)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41700)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41700) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41700) 
(pid=41700) During handling of the above exception, another exception occurred:
(pid=41700) 
(pid=41700) Traceback (most recent call last):
(pid=41700)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41700)     job_id=None)
(pid=41700)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41700)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41700)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41700)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41700) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41671) Traceback (most recent call last):
(pid=41671)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41671)     ray.worker.global_worker.main_loop()
(pid=41671)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41671)     task = self._get_next_task_from_raylet()
(pid=41671)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41671)     task = self.raylet_client.get_task()
(pid=41671)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41671)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41671) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41671) 
(pid=41671) During handling of the above exception, another exception occurred:
(pid=41671) 
(pid=41671) Traceback (most recent call last):
(pid=41671)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41671)     job_id=None)
(pid=41671)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41671)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41671)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41671)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41671) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41721) Traceback (most recent call last):
(pid=41721)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41721)     ray.worker.global_worker.main_loop()
(pid=41721)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41721)     task = self._get_next_task_from_raylet()
(pid=41721)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41721)     task = self.raylet_client.get_task()
(pid=41721)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41721)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41721) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41721) 
(pid=41721) During handling of the above exception, another exception occurred:
(pid=41721) 
(pid=41721) Traceback (most recent call last):
(pid=41721)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41721)     job_id=None)
(pid=41721)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41721)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41721)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41721)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41721) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41710) Traceback (most recent call last):
(pid=41710)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41710)     ray.worker.global_worker.main_loop()
(pid=41710)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41710)     task = self._get_next_task_from_raylet()
(pid=41710)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41710)     task = self.raylet_client.get_task()
(pid=41710)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41710)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41710) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41710) 
(pid=41710) During handling of the above exception, another exception occurred:
(pid=41710) 
(pid=41710) Traceback (most recent call last):
(pid=41710)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41710)     job_id=None)
(pid=41710)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41710)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41710)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41710)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41710) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41685) Traceback (most recent call last):
(pid=41685)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41685)     ray.worker.global_worker.main_loop()
(pid=41685)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41685)     task = self._get_next_task_from_raylet()
(pid=41685)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41685)     task = self.raylet_client.get_task()
(pid=41685)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41685)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41685) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41685) 
(pid=41685) During handling of the above exception, another exception occurred:
(pid=41685) 
(pid=41685) Traceback (most recent call last):
(pid=41685)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41685)     job_id=None)
(pid=41685)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41685)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41685)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41685)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41685) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41722) Traceback (most recent call last):
(pid=41722)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41722)     ray.worker.global_worker.main_loop()
(pid=41722)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41722)     task = self._get_next_task_from_raylet()
(pid=41722)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41722)     task = self.raylet_client.get_task()
(pid=41722)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41722)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41722) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41722) 
(pid=41722) During handling of the above exception, another exception occurred:
(pid=41722) 
(pid=41722) Traceback (most recent call last):
(pid=41722)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41722)     job_id=None)
(pid=41722)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41722)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41722)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41722)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41722) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41733) Traceback (most recent call last):
(pid=41733)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41733)     ray.worker.global_worker.main_loop()
(pid=41733)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41733)     task = self._get_next_task_from_raylet()
(pid=41733)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41733)     task = self.raylet_client.get_task()
(pid=41733)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41733)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41733) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41733) 
(pid=41733) During handling of the above exception, another exception occurred:
(pid=41733) 
(pid=41733) Traceback (most recent call last):
(pid=41733)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41733)     job_id=None)
(pid=41733)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41733)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41733)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41733)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41733) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41682) Traceback (most recent call last):
(pid=41682)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41682)     ray.worker.global_worker.main_loop()
(pid=41682)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41682)     task = self._get_next_task_from_raylet()
(pid=41682)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41682)     task = self.raylet_client.get_task()
(pid=41682)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41682)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41682) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41682) 
(pid=41682) During handling of the above exception, another exception occurred:
(pid=41682) 
(pid=41682) Traceback (most recent call last):
(pid=41682)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41682)     job_id=None)
(pid=41682)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41682)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41682)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41682)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41682) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41717) Traceback (most recent call last):
(pid=41717)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41717)     ray.worker.global_worker.main_loop()
(pid=41717)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41717)     task = self._get_next_task_from_raylet()
(pid=41717)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41717)     task = self.raylet_client.get_task()
(pid=41717)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41717)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41717) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41717) 
(pid=41717) During handling of the above exception, another exception occurred:
(pid=41717) 
(pid=41717) Traceback (most recent call last):
(pid=41717)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41717)     job_id=None)
(pid=41717)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41717)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41717)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41717)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41717) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41688) Traceback (most recent call last):
(pid=41688)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41688)     ray.worker.global_worker.main_loop()
(pid=41688)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41688)     task = self._get_next_task_from_raylet()
(pid=41688)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41688)     task = self.raylet_client.get_task()
(pid=41688)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41688)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41688) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41688) 
(pid=41688) During handling of the above exception, another exception occurred:
(pid=41688) 
(pid=41688) Traceback (most recent call last):
(pid=41688)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41688)     job_id=None)
(pid=41688)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41688)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41688)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41688)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41688) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41648) Traceback (most recent call last):
(pid=41648)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41648)     ray.worker.global_worker.main_loop()
(pid=41648)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41648)     task = self._get_next_task_from_raylet()
(pid=41648)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41648)     task = self.raylet_client.get_task()
(pid=41648)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41648)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41648) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41648) 
(pid=41648) During handling of the above exception, another exception occurred:
(pid=41648) 
(pid=41648) Traceback (most recent call last):
(pid=41648)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41648)     job_id=None)
(pid=41648)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41648)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41648)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41648)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41648) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41711) Traceback (most recent call last):
(pid=41711)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41711)     ray.worker.global_worker.main_loop()
(pid=41711)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41711)     task = self._get_next_task_from_raylet()
(pid=41711)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41711)     task = self.raylet_client.get_task()
(pid=41711)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41711)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41711) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41711) 
(pid=41711) During handling of the above exception, another exception occurred:
(pid=41711) 
(pid=41711) Traceback (most recent call last):
(pid=41711)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41711)     job_id=None)
(pid=41711)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41711)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41711)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41711)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41711) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41678) Traceback (most recent call last):
(pid=41678)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41678)     ray.worker.global_worker.main_loop()
(pid=41678)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41678)     task = self._get_next_task_from_raylet()
(pid=41678)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41678)     task = self.raylet_client.get_task()
(pid=41678)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41678)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41678) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41678) 
(pid=41678) During handling of the above exception, another exception occurred:
(pid=41678) 
(pid=41678) Traceback (most recent call last):
(pid=41678)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41678)     job_id=None)
(pid=41678)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41678)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41678)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41678)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41678) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41662) Traceback (most recent call last):
(pid=41662)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41662)     ray.worker.global_worker.main_loop()
(pid=41662)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41662)     task = self._get_next_task_from_raylet()
(pid=41662)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41662)     task = self.raylet_client.get_task()
(pid=41662)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41662)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41662) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41662) 
(pid=41662) During handling of the above exception, another exception occurred:
(pid=41662) 
(pid=41662) Traceback (most recent call last):
(pid=41662)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41662)     job_id=None)
(pid=41662)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41662)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41662)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41662)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41662) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41670) Traceback (most recent call last):
(pid=41670)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41670)     ray.worker.global_worker.main_loop()
(pid=41670)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41670)     task = self._get_next_task_from_raylet()
(pid=41670)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41670)     task = self.raylet_client.get_task()
(pid=41670)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41670)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41670) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41670) 
(pid=41670) During handling of the above exception, another exception occurred:
(pid=41670) 
(pid=41670) Traceback (most recent call last):
(pid=41670)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41670)     job_id=None)
(pid=41670)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41670)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41670)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41670)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41670) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41667) Traceback (most recent call last):
(pid=41667)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41667)     ray.worker.global_worker.main_loop()
(pid=41667)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41667)     task = self._get_next_task_from_raylet()
(pid=41667)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41667)     task = self.raylet_client.get_task()
(pid=41667)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41667)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41667) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41667) 
(pid=41667) During handling of the above exception, another exception occurred:
(pid=41667) 
(pid=41667) Traceback (most recent call last):
(pid=41667)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41667)     job_id=None)
(pid=41667)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41667)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41667)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41667)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41667) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41714) Traceback (most recent call last):
(pid=41714)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41714)     ray.worker.global_worker.main_loop()
(pid=41714)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41714)     task = self._get_next_task_from_raylet()
(pid=41714)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41714)     task = self.raylet_client.get_task()
(pid=41714)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41714)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41714) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41714) 
(pid=41714) During handling of the above exception, another exception occurred:
(pid=41714) 
(pid=41714) Traceback (most recent call last):
(pid=41714)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41714)     job_id=None)
(pid=41714)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41714)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41714)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41714)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41714) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41668) Traceback (most recent call last):
(pid=41668)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41668)     ray.worker.global_worker.main_loop()
(pid=41668)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41668)     task = self._get_next_task_from_raylet()
(pid=41668)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41668)     task = self.raylet_client.get_task()
(pid=41668)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41668)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41668) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41668) 
(pid=41668) During handling of the above exception, another exception occurred:
(pid=41668) 
(pid=41668) Traceback (most recent call last):
(pid=41668)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41668)     job_id=None)
(pid=41668)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41668)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41668)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41668)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41668) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41726) Traceback (most recent call last):
(pid=41726)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41726)     ray.worker.global_worker.main_loop()
(pid=41726)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41726)     task = self._get_next_task_from_raylet()
(pid=41726)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41726)     task = self.raylet_client.get_task()
(pid=41726)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41726)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41726) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41726) 
(pid=41726) During handling of the above exception, another exception occurred:
(pid=41726) 
(pid=41726) Traceback (most recent call last):
(pid=41726)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41726)     job_id=None)
(pid=41726)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41726)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41726)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41726)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41726) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41684) Traceback (most recent call last):
(pid=41684)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41684)     ray.worker.global_worker.main_loop()
(pid=41684)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41684)     task = self._get_next_task_from_raylet()
(pid=41684)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41684)     task = self.raylet_client.get_task()
(pid=41684)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41684)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41684) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41684) 
(pid=41684) During handling of the above exception, another exception occurred:
(pid=41684) 
(pid=41684) Traceback (most recent call last):
(pid=41684)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41684)     job_id=None)
(pid=41684)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41684)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41684)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41684)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41684) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41683) Traceback (most recent call last):
(pid=41683)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41683)     ray.worker.global_worker.main_loop()
(pid=41683)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41683)     task = self._get_next_task_from_raylet()
(pid=41683)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41683)     task = self.raylet_client.get_task()
(pid=41683)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41683)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41683) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41683) 
(pid=41683) During handling of the above exception, another exception occurred:
(pid=41683) 
(pid=41683) Traceback (most recent call last):
(pid=41683)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41683)     job_id=None)
(pid=41683)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41683)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41683)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41683)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41683) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41727) Traceback (most recent call last):
(pid=41727)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41727)     ray.worker.global_worker.main_loop()
(pid=41727)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41727)     task = self._get_next_task_from_raylet()
(pid=41727)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41727)     task = self.raylet_client.get_task()
(pid=41727)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41727)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41727) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41727) 
(pid=41727) During handling of the above exception, another exception occurred:
(pid=41727) 
(pid=41727) Traceback (most recent call last):
(pid=41727)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41727)     job_id=None)
(pid=41727)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41727)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41727)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41727)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41727) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41675) Traceback (most recent call last):
(pid=41675)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41675)     ray.worker.global_worker.main_loop()
(pid=41675)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41675)     task = self._get_next_task_from_raylet()
(pid=41675)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41675)     task = self.raylet_client.get_task()
(pid=41675)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41675)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41675) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41675) 
(pid=41675) During handling of the above exception, another exception occurred:
(pid=41675) 
(pid=41675) Traceback (most recent call last):
(pid=41675)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41675)     job_id=None)
(pid=41675)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41675)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41675)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41675)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41675) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41709) Traceback (most recent call last):
(pid=41709)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41709)     ray.worker.global_worker.main_loop()
(pid=41709)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41709)     task = self._get_next_task_from_raylet()
(pid=41709)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41709)     task = self.raylet_client.get_task()
(pid=41709)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41709)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41709) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41709) 
(pid=41709) During handling of the above exception, another exception occurred:
(pid=41709) 
(pid=41709) Traceback (most recent call last):
(pid=41709)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41709)     job_id=None)
(pid=41709)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41709)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41709)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41709)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41709) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41665) Traceback (most recent call last):
(pid=41665)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41665)     ray.worker.global_worker.main_loop()
(pid=41665)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41665)     task = self._get_next_task_from_raylet()
(pid=41665)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41665)     task = self.raylet_client.get_task()
(pid=41665)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41665)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41665) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41665) 
(pid=41665) During handling of the above exception, another exception occurred:
(pid=41665) 
(pid=41665) Traceback (most recent call last):
(pid=41665)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41665)     job_id=None)
(pid=41665)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41665)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41665)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41665)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41665) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41659) Traceback (most recent call last):
(pid=41659)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41659)     ray.worker.global_worker.main_loop()
(pid=41659)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41659)     task = self._get_next_task_from_raylet()
(pid=41659)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41659)     task = self.raylet_client.get_task()
(pid=41659)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41659)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41659) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41659) 
(pid=41659) During handling of the above exception, another exception occurred:
(pid=41659) 
(pid=41659) Traceback (most recent call last):
(pid=41659)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41659)     job_id=None)
(pid=41659)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41659)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41659)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41659)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41659) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41706) Traceback (most recent call last):
(pid=41706)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41706)     ray.worker.global_worker.main_loop()
(pid=41706)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41706)     task = self._get_next_task_from_raylet()
(pid=41706)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41706)     task = self.raylet_client.get_task()
(pid=41706)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41706)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41706) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41706) 
(pid=41706) During handling of the above exception, another exception occurred:
(pid=41706) 
(pid=41706) Traceback (most recent call last):
(pid=41706)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41706)     job_id=None)
(pid=41706)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41706)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41706)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41706)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41706) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41694) Traceback (most recent call last):
(pid=41694)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41694)     ray.worker.global_worker.main_loop()
(pid=41694)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41694)     task = self._get_next_task_from_raylet()
(pid=41694)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41694)     task = self.raylet_client.get_task()
(pid=41694)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41694)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41694) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41694) 
(pid=41694) During handling of the above exception, another exception occurred:
(pid=41694) 
(pid=41694) Traceback (most recent call last):
(pid=41694)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41694)     job_id=None)
(pid=41694)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41694)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41694)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41694)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41694) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41691) Traceback (most recent call last):
(pid=41691)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41691)     ray.worker.global_worker.main_loop()
(pid=41691)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41691)     task = self._get_next_task_from_raylet()
(pid=41691)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41691)     task = self.raylet_client.get_task()
(pid=41691)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41691)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41691) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41691) 
(pid=41691) During handling of the above exception, another exception occurred:
(pid=41691) 
(pid=41691) Traceback (most recent call last):
(pid=41691)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41691)     job_id=None)
(pid=41691)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41691)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41691)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41691)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41691) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41673) Traceback (most recent call last):
(pid=41673)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41673)     ray.worker.global_worker.main_loop()
(pid=41673)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41673)     task = self._get_next_task_from_raylet()
(pid=41673)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41673)     task = self.raylet_client.get_task()
(pid=41673)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41673)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41673) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41673) 
(pid=41673) During handling of the above exception, another exception occurred:
(pid=41673) 
(pid=41673) Traceback (most recent call last):
(pid=41673)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41673)     job_id=None)
(pid=41673)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41673)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41673)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41673)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41673) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41713) Traceback (most recent call last):
(pid=41713)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41713)     ray.worker.global_worker.main_loop()
(pid=41713)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41713)     task = self._get_next_task_from_raylet()
(pid=41713)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41713)     task = self.raylet_client.get_task()
(pid=41713)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41713)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41713) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41713) 
(pid=41713) During handling of the above exception, another exception occurred:
(pid=41713) 
(pid=41713) Traceback (most recent call last):
(pid=41713)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41713)     job_id=None)
(pid=41713)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41713)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41713)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41713)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41713) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41698) Traceback (most recent call last):
(pid=41698)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41698)     ray.worker.global_worker.main_loop()
(pid=41698)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41698)     task = self._get_next_task_from_raylet()
(pid=41698)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41698)     task = self.raylet_client.get_task()
(pid=41698)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41698)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41698) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41698) 
(pid=41698) During handling of the above exception, another exception occurred:
(pid=41698) 
(pid=41698) Traceback (most recent call last):
(pid=41698)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41698)     job_id=None)
(pid=41698)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41698)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41698)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41698)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41698) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41695) Traceback (most recent call last):
(pid=41695)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41695)     ray.worker.global_worker.main_loop()
(pid=41695)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41695)     task = self._get_next_task_from_raylet()
(pid=41695)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41695)     task = self.raylet_client.get_task()
(pid=41695)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41695)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41695) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41695) 
(pid=41695) During handling of the above exception, another exception occurred:
(pid=41695) 
(pid=41695) Traceback (most recent call last):
(pid=41695)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41695)     job_id=None)
(pid=41695)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41695)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41695)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41695)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41695) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41707) Traceback (most recent call last):
(pid=41707)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41707)     ray.worker.global_worker.main_loop()
(pid=41707)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41707)     task = self._get_next_task_from_raylet()
(pid=41707)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41707)     task = self.raylet_client.get_task()
(pid=41707)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41707)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41707) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41707) 
(pid=41707) During handling of the above exception, another exception occurred:
(pid=41707) 
(pid=41707) Traceback (most recent call last):
(pid=41707)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41707)     job_id=None)
(pid=41707)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41707)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41707)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41707)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41707) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41687) Traceback (most recent call last):
(pid=41687)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41687)     ray.worker.global_worker.main_loop()
(pid=41687)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41687)     task = self._get_next_task_from_raylet()
(pid=41687)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41687)     task = self.raylet_client.get_task()
(pid=41687)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41687)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41687) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41687) 
(pid=41687) During handling of the above exception, another exception occurred:
(pid=41687) 
(pid=41687) Traceback (most recent call last):
(pid=41687)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41687)     job_id=None)
(pid=41687)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41687)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41687)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41687)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41687) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41647) Traceback (most recent call last):
(pid=41647)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41647)     ray.worker.global_worker.main_loop()
(pid=41647)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41647)     task = self._get_next_task_from_raylet()
(pid=41647)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41647)     task = self.raylet_client.get_task()
(pid=41647)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41647)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41647) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41647) 
(pid=41647) During handling of the above exception, another exception occurred:
(pid=41647) 
(pid=41647) Traceback (most recent call last):
(pid=41647)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41647)     job_id=None)
(pid=41647)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41647)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41647)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41647)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41647) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41664) Traceback (most recent call last):
(pid=41664)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41664)     ray.worker.global_worker.main_loop()
(pid=41664)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41664)     task = self._get_next_task_from_raylet()
(pid=41664)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41664)     task = self.raylet_client.get_task()
(pid=41664)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41664)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41664) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41664) 
(pid=41664) During handling of the above exception, another exception occurred:
(pid=41664) 
(pid=41664) Traceback (most recent call last):
(pid=41664)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41664)     job_id=None)
(pid=41664)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41664)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41664)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41664)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41664) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41704) Traceback (most recent call last):
(pid=41704)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41704)     ray.worker.global_worker.main_loop()
(pid=41704)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41704)     task = self._get_next_task_from_raylet()
(pid=41704)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41704)     task = self.raylet_client.get_task()
(pid=41704)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41704)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41704) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41704) 
(pid=41704) During handling of the above exception, another exception occurred:
(pid=41704) 
(pid=41704) Traceback (most recent call last):
(pid=41704)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41704)     job_id=None)
(pid=41704)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41704)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41704)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41704)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41704) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41663) Traceback (most recent call last):
(pid=41663)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41663)     ray.worker.global_worker.main_loop()
(pid=41663)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41663)     task = self._get_next_task_from_raylet()
(pid=41663)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41663)     task = self.raylet_client.get_task()
(pid=41663)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41663)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41663) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41663) 
(pid=41663) During handling of the above exception, another exception occurred:
(pid=41663) 
(pid=41663) Traceback (most recent call last):
(pid=41663)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41663)     job_id=None)
(pid=41663)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41663)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41663)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41663)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41663) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41689) Traceback (most recent call last):
(pid=41689)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41689)     ray.worker.global_worker.main_loop()
(pid=41689)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41689)     task = self._get_next_task_from_raylet()
(pid=41689)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41689)     task = self.raylet_client.get_task()
(pid=41689)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41689)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41689) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41689) 
(pid=41689) During handling of the above exception, another exception occurred:
(pid=41689) 
(pid=41689) Traceback (most recent call last):
(pid=41689)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41689)     job_id=None)
(pid=41689)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41689)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41689)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41689)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41689) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41719) Traceback (most recent call last):
(pid=41719)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41719)     ray.worker.global_worker.main_loop()
(pid=41719)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41719)     task = self._get_next_task_from_raylet()
(pid=41719)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41719)     task = self.raylet_client.get_task()
(pid=41719)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41719)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41719) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41719) 
(pid=41719) During handling of the above exception, another exception occurred:
(pid=41719) 
(pid=41719) Traceback (most recent call last):
(pid=41719)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41719)     job_id=None)
(pid=41719)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41719)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41719)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41719)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41719) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41661) Traceback (most recent call last):
(pid=41661)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41661)     ray.worker.global_worker.main_loop()
(pid=41661)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41661)     task = self._get_next_task_from_raylet()
(pid=41661)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41661)     task = self.raylet_client.get_task()
(pid=41661)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41661)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41661) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41661) 
(pid=41661) During handling of the above exception, another exception occurred:
(pid=41661) 
(pid=41661) Traceback (most recent call last):
(pid=41661)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41661)     job_id=None)
(pid=41661)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41661)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41661)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41661)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41661) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41653) Traceback (most recent call last):
(pid=41653)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41653)     ray.worker.global_worker.main_loop()
(pid=41653)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41653)     task = self._get_next_task_from_raylet()
(pid=41653)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41653)     task = self.raylet_client.get_task()
(pid=41653)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41653)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41653) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41653) 
(pid=41653) During handling of the above exception, another exception occurred:
(pid=41653) 
(pid=41653) Traceback (most recent call last):
(pid=41653)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41653)     job_id=None)
(pid=41653)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41653)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41653)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41653)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41653) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41699) Traceback (most recent call last):
(pid=41699)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41699)     ray.worker.global_worker.main_loop()
(pid=41699)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41699)     task = self._get_next_task_from_raylet()
(pid=41699)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41699)     task = self.raylet_client.get_task()
(pid=41699)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41699)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41699) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41699) 
(pid=41699) During handling of the above exception, another exception occurred:
(pid=41699) 
(pid=41699) Traceback (most recent call last):
(pid=41699)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41699)     job_id=None)
(pid=41699)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41699)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41699)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41699)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41699) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41690) Traceback (most recent call last):
(pid=41690)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41690)     ray.worker.global_worker.main_loop()
(pid=41690)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41690)     task = self._get_next_task_from_raylet()
(pid=41690)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41690)     task = self.raylet_client.get_task()
(pid=41690)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41690)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41690) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41690) 
(pid=41690) During handling of the above exception, another exception occurred:
(pid=41690) 
(pid=41690) Traceback (most recent call last):
(pid=41690)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41690)     job_id=None)
(pid=41690)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41690)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41690)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41690)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41690) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41674) Traceback (most recent call last):
(pid=41674)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41674)     ray.worker.global_worker.main_loop()
(pid=41674)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41674)     task = self._get_next_task_from_raylet()
(pid=41674)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41674)     task = self.raylet_client.get_task()
(pid=41674)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41674)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41674) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41674) 
(pid=41674) During handling of the above exception, another exception occurred:
(pid=41674) 
(pid=41674) Traceback (most recent call last):
(pid=41674)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41674)     job_id=None)
(pid=41674)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41674)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41674)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41674)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41674) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41652) Traceback (most recent call last):
(pid=41652)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41652)     ray.worker.global_worker.main_loop()
(pid=41652)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41652)     task = self._get_next_task_from_raylet()
(pid=41652)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41652)     task = self.raylet_client.get_task()
(pid=41652)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41652)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41652) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41652) 
(pid=41652) During handling of the above exception, another exception occurred:
(pid=41652) 
(pid=41652) Traceback (most recent call last):
(pid=41652)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41652)     job_id=None)
(pid=41652)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41652)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41652)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41652)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41652) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41696) Traceback (most recent call last):
(pid=41696)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41696)     ray.worker.global_worker.main_loop()
(pid=41696)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41696)     task = self._get_next_task_from_raylet()
(pid=41696)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41696)     task = self.raylet_client.get_task()
(pid=41696)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41696) WARNING: Logging before InitGoogleLogging() is written to STDERR
(pid=41696) E1118 16:58:52.885603 41973 raylet_client.cc:343] IOError: [RayletClient] Connection closed unexpectedly. [RayletClient] Failed to push profile events.
(pid=41696)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41696) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41696) 
(pid=41696) During handling of the above exception, another exception occurred:
(pid=41696) 
(pid=41696) Traceback (most recent call last):
(pid=41696)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41696)     job_id=None)
(pid=41696)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41696)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41696)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41696)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41696) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41731) Traceback (most recent call last):
(pid=41731)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41731)     ray.worker.global_worker.main_loop()
(pid=41731)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41731)     task = self._get_next_task_from_raylet()
(pid=41731)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41731)     task = self.raylet_client.get_task()
(pid=41731)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41731)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41731) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41731) 
(pid=41731) During handling of the above exception, another exception occurred:
(pid=41731) 
(pid=41731) Traceback (most recent call last):
(pid=41731)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41731)     job_id=None)
(pid=41731)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41731)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41731)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41731)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41731) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41655) Traceback (most recent call last):
(pid=41655)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41655)     ray.worker.global_worker.main_loop()
(pid=41655)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41655)     task = self._get_next_task_from_raylet()
(pid=41655)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41655)     task = self.raylet_client.get_task()
(pid=41655)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41655)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41655) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41655) 
(pid=41655) During handling of the above exception, another exception occurred:
(pid=41655) 
(pid=41655) Traceback (most recent call last):
(pid=41655)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41655)     job_id=None)
(pid=41655)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41655)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41655)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41655)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41655) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41680) Traceback (most recent call last):
(pid=41680)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41680)     ray.worker.global_worker.main_loop()
(pid=41680)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41680)     task = self._get_next_task_from_raylet()
(pid=41680)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41680)     task = self.raylet_client.get_task()
(pid=41680)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41680)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41680) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41680) 
(pid=41680) During handling of the above exception, another exception occurred:
(pid=41680) 
(pid=41680) Traceback (most recent call last):
(pid=41680)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41680)     job_id=None)
(pid=41680)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41680)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41680)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41680)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41680) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41686) Traceback (most recent call last):
(pid=41686)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41686)     ray.worker.global_worker.main_loop()
(pid=41686)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41686)     task = self._get_next_task_from_raylet()
(pid=41686)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41686)     task = self.raylet_client.get_task()
(pid=41686)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41686)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41686) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41686) 
(pid=41686) During handling of the above exception, another exception occurred:
(pid=41686) 
(pid=41686) Traceback (most recent call last):
(pid=41686)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41686)     job_id=None)
(pid=41686)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41686)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41686)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41686)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41686) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41681) Traceback (most recent call last):
(pid=41681)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41681)     ray.worker.global_worker.main_loop()
(pid=41681)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41681)     task = self._get_next_task_from_raylet()
(pid=41681)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41681)     task = self.raylet_client.get_task()
(pid=41681)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41681)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41681) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41681) 
(pid=41681) During handling of the above exception, another exception occurred:
(pid=41681) 
(pid=41681) Traceback (most recent call last):
(pid=41681)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41681)     job_id=None)
(pid=41681)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41681)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41681)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41681)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41681) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41679) Traceback (most recent call last):
(pid=41679)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41679)     ray.worker.global_worker.main_loop()
(pid=41679)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41679)     task = self._get_next_task_from_raylet()
(pid=41679)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41679)     task = self.raylet_client.get_task()
(pid=41679)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41679)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41679) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41679) 
(pid=41679) During handling of the above exception, another exception occurred:
(pid=41679) 
(pid=41679) Traceback (most recent call last):
(pid=41679)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41679)     job_id=None)
(pid=41679)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41679)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41679)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41679)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41679) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41697) Traceback (most recent call last):
(pid=41697)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41697)     ray.worker.global_worker.main_loop()
(pid=41697)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41697)     task = self._get_next_task_from_raylet()
(pid=41697)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41697)     task = self.raylet_client.get_task()
(pid=41697)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41697)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41697) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41697) 
(pid=41697) During handling of the above exception, another exception occurred:
(pid=41697) 
(pid=41697) Traceback (most recent call last):
(pid=41697)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41697)     job_id=None)
(pid=41697)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41697)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41697)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41697)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41697) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41720) Traceback (most recent call last):
(pid=41720)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41720)     ray.worker.global_worker.main_loop()
(pid=41720)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41720)     task = self._get_next_task_from_raylet()
(pid=41720)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41720)     task = self.raylet_client.get_task()
(pid=41720)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41720)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41720) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41720) 
(pid=41720) During handling of the above exception, another exception occurred:
(pid=41720) 
(pid=41720) Traceback (most recent call last):
(pid=41720)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41720)     job_id=None)
(pid=41720)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41720)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41720)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41720)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41720) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41677) Traceback (most recent call last):
(pid=41677)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41677)     ray.worker.global_worker.main_loop()
(pid=41677)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41677)     task = self._get_next_task_from_raylet()
(pid=41677)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41677)     task = self.raylet_client.get_task()
(pid=41677)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41677)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41677) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41677) 
(pid=41677) During handling of the above exception, another exception occurred:
(pid=41677) 
(pid=41677) Traceback (most recent call last):
(pid=41677)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41677)     job_id=None)
(pid=41677)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41677)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41677)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41677)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41677) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41724) Traceback (most recent call last):
(pid=41724)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41724)     ray.worker.global_worker.main_loop()
(pid=41724)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41724)     task = self._get_next_task_from_raylet()
(pid=41724)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41724)     task = self.raylet_client.get_task()
(pid=41724)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41724)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41724) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41724) 
(pid=41724) During handling of the above exception, another exception occurred:
(pid=41724) 
(pid=41724) Traceback (most recent call last):
(pid=41724)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41724)     job_id=None)
(pid=41724)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41724)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41724)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41724)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41724) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41716) Traceback (most recent call last):
(pid=41716)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41716)     ray.worker.global_worker.main_loop()
(pid=41716)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41716)     task = self._get_next_task_from_raylet()
(pid=41716)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41716)     task = self.raylet_client.get_task()
(pid=41716)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41716)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41716) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41716) 
(pid=41716) During handling of the above exception, another exception occurred:
(pid=41716) 
(pid=41716) Traceback (most recent call last):
(pid=41716)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41716)     job_id=None)
(pid=41716)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41716)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41716)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41716)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41716) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41649) Traceback (most recent call last):
(pid=41649)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41649)     ray.worker.global_worker.main_loop()
(pid=41649)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41649)     task = self._get_next_task_from_raylet()
(pid=41649)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41649)     task = self.raylet_client.get_task()
(pid=41649)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41649)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41649) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41649) 
(pid=41649) During handling of the above exception, another exception occurred:
(pid=41649) 
(pid=41649) Traceback (most recent call last):
(pid=41649)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41649)     job_id=None)
(pid=41649)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41649)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41649)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41649)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41649) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41676) Traceback (most recent call last):
(pid=41676)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41676)     ray.worker.global_worker.main_loop()
(pid=41676)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41676)     task = self._get_next_task_from_raylet()
(pid=41676)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41676)     task = self.raylet_client.get_task()
(pid=41676)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41676)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41676) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41676) 
(pid=41676) During handling of the above exception, another exception occurred:
(pid=41676) 
(pid=41676) Traceback (most recent call last):
(pid=41676)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41676)     job_id=None)
(pid=41676)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41676)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41676)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41676)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41676) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41715) Traceback (most recent call last):
(pid=41715)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41715)     ray.worker.global_worker.main_loop()
(pid=41715)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41715)     task = self._get_next_task_from_raylet()
(pid=41715)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41715)     task = self.raylet_client.get_task()
(pid=41715)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41715)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41715) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41715) 
(pid=41715) During handling of the above exception, another exception occurred:
(pid=41715) 
(pid=41715) Traceback (most recent call last):
(pid=41715)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41715)     job_id=None)
(pid=41715)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41715)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41715)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41715)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41715) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41725) Traceback (most recent call last):
(pid=41725)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41725)     ray.worker.global_worker.main_loop()
(pid=41725)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41725)     task = self._get_next_task_from_raylet()
(pid=41725)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41725)     task = self.raylet_client.get_task()
(pid=41725)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41725)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41725) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41725) 
(pid=41725) During handling of the above exception, another exception occurred:
(pid=41725) 
(pid=41725) Traceback (most recent call last):
(pid=41725)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41725)     job_id=None)
(pid=41725)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41725)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41725)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41725)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41725) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41693) Traceback (most recent call last):
(pid=41693)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41693)     ray.worker.global_worker.main_loop()
(pid=41693)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41693)     task = self._get_next_task_from_raylet()
(pid=41693)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41693)     task = self.raylet_client.get_task()
(pid=41693)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41693)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41693) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41693) 
(pid=41693) During handling of the above exception, another exception occurred:
(pid=41693) 
(pid=41693) Traceback (most recent call last):
(pid=41693)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41693)     job_id=None)
(pid=41693)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41693)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41693)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41693)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41693) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41729) Traceback (most recent call last):
(pid=41729)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41729)     ray.worker.global_worker.main_loop()
(pid=41729)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41729)     task = self._get_next_task_from_raylet()
(pid=41729)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41729)     task = self.raylet_client.get_task()
(pid=41729)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41729)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41729) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41729) 
(pid=41729) During handling of the above exception, another exception occurred:
(pid=41729) 
(pid=41729) Traceback (most recent call last):
(pid=41729)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41729)     job_id=None)
(pid=41729)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41729)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41729)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41729)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41729) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41701) Traceback (most recent call last):
(pid=41701)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41701)     ray.worker.global_worker.main_loop()
(pid=41701)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41701)     task = self._get_next_task_from_raylet()
(pid=41701)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41701)     task = self.raylet_client.get_task()
(pid=41701)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41701)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41701) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41701) 
(pid=41701) During handling of the above exception, another exception occurred:
(pid=41701) 
(pid=41701) Traceback (most recent call last):
(pid=41701)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41701)     job_id=None)
(pid=41701)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41701)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41701)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41701)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41701) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41669) Traceback (most recent call last):
(pid=41669)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41669)     ray.worker.global_worker.main_loop()
(pid=41669)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41669)     task = self._get_next_task_from_raylet()
(pid=41669)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41669)     task = self.raylet_client.get_task()
(pid=41669)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41669)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41669) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41669) 
(pid=41669) During handling of the above exception, another exception occurred:
(pid=41669) 
(pid=41669) Traceback (most recent call last):
(pid=41669)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41669)     job_id=None)
(pid=41669)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41669)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41669)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41669)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41669) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41718) Traceback (most recent call last):
(pid=41718)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41718)     ray.worker.global_worker.main_loop()
(pid=41718)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41718)     task = self._get_next_task_from_raylet()
(pid=41718)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41718)     task = self.raylet_client.get_task()
(pid=41718)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41718)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41718) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41718) 
(pid=41718) During handling of the above exception, another exception occurred:
(pid=41718) 
(pid=41718) Traceback (most recent call last):
(pid=41718)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41718)     job_id=None)
(pid=41718)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41718)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41718)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41718)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41718) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41732) Traceback (most recent call last):
(pid=41732)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41732)     ray.worker.global_worker.main_loop()
(pid=41732)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41732)     task = self._get_next_task_from_raylet()
(pid=41732)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41732)     task = self.raylet_client.get_task()
(pid=41732)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41732)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41732) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41732) 
(pid=41732) During handling of the above exception, another exception occurred:
(pid=41732) 
(pid=41732) Traceback (most recent call last):
(pid=41732)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41732)     job_id=None)
(pid=41732)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41732)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41732)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41732)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41732) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41692) Traceback (most recent call last):
(pid=41692)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41692)     ray.worker.global_worker.main_loop()
(pid=41692)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41692)     task = self._get_next_task_from_raylet()
(pid=41692)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41692)     task = self.raylet_client.get_task()
(pid=41692)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41692)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41692) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41692) 
(pid=41692) During handling of the above exception, another exception occurred:
(pid=41692) 
(pid=41692) Traceback (most recent call last):
(pid=41692)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41692)     job_id=None)
(pid=41692)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41692)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41692)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41692)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41692) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41730) Traceback (most recent call last):
(pid=41730)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41730)     ray.worker.global_worker.main_loop()
(pid=41730)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41730)     task = self._get_next_task_from_raylet()
(pid=41730)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41730)     task = self.raylet_client.get_task()
(pid=41730)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41730)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41730) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Raylet connection closed.
(pid=41730) 
(pid=41730) During handling of the above exception, another exception occurred:
(pid=41730) 
(pid=41730) Traceback (most recent call last):
(pid=41730)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41730)     job_id=None)
(pid=41730)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41730)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41730)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41730)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41730) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41702) WARNING: Logging before InitGoogleLogging() is written to STDERR
(pid=41702) E1118 16:58:53.084489 41928 raylet_client.cc:343] IOError: [RayletClient] Connection closed unexpectedly. [RayletClient] Failed to push profile events.
(pid=41702) Traceback (most recent call last):
(pid=41702)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 98, in <module>
(pid=41702)     ray.worker.global_worker.main_loop()
(pid=41702)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1087, in main_loop
(pid=41702)     task = self._get_next_task_from_raylet()
(pid=41702)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py", line 1070, in _get_next_task_from_raylet
(pid=41702)     task = self.raylet_client.get_task()
(pid=41702)   File "python/ray/_raylet.pyx", line 247, in ray._raylet.RayletClient.get_task
(pid=41702)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41702) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41702) 
(pid=41702) During handling of the above exception, another exception occurred:
(pid=41702) 
(pid=41702) Traceback (most recent call last):
(pid=41702)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/workers/default_worker.py", line 105, in <module>
(pid=41702)     job_id=None)
(pid=41702)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py", line 68, in push_error_to_driver
(pid=41702)     worker.raylet_client.push_error(job_id, error_type, message, time.time())
(pid=41702)   File "python/ray/_raylet.pyx", line 300, in ray._raylet.RayletClient.push_error
(pid=41702)   File "python/ray/_raylet.pyx", line 61, in ray._raylet.check_status
(pid=41702) ray.exceptions.RayletError: The Raylet died with this message: [RayletClient] Connection closed unexpectedly.
(pid=41698) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f7b822df730>
(pid=41698) Traceback (most recent call last):
(pid=41698)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41698) TypeError: 'NoneType' object is not callable
(pid=41690) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fb85b9df730>
(pid=41690) Traceback (most recent call last):
(pid=41690)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41690) TypeError: 'NoneType' object is not callable
(pid=41673) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fb013f27730>
(pid=41673) Traceback (most recent call last):
(pid=41673)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41673) TypeError: 'NoneType' object is not callable
(pid=41728) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fd2601d7730>
(pid=41728) Traceback (most recent call last):
(pid=41728)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41728) TypeError: 'NoneType' object is not callable
(pid=41671) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fa6dfb63730>
(pid=41671) Traceback (most recent call last):
(pid=41671)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41671) TypeError: 'NoneType' object is not callable
(pid=41714) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fca19ea3730>
(pid=41714) Traceback (most recent call last):
(pid=41714)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41714) TypeError: 'NoneType' object is not callable
(pid=41646) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f6ad1156730>
(pid=41646) Traceback (most recent call last):
(pid=41646)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41646) TypeError: 'NoneType' object is not callable
(pid=41687) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fb9f8f14730>
(pid=41687) Traceback (most recent call last):
(pid=41687)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41687) TypeError: 'NoneType' object is not callable
(pid=41663) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fd6ea2df730>
(pid=41663) Traceback (most recent call last):
(pid=41663)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41663) TypeError: 'NoneType' object is not callable
(pid=41713) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fb8028b6730>
(pid=41713) Traceback (most recent call last):
(pid=41713)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41713) TypeError: 'NoneType' object is not callable
---------------------------------------------------------------------------
ArrowIOError                              Traceback (most recent call last)
<ipython-input-10-8198f8a7bbcb> in <module>
----> 1 scaled_feats = StandardScaler().fit_transform(mat.values)

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/modin/pandas/base.py in __getattribute__(self, item)
   3431         ]
   3432         if item not in default_behaviors:
-> 3433             method = object.__getattribute__(self, item)
   3434             is_callable = callable(method)
   3435             # We default to pandas on empty DataFrames. This avoids a large amount of

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/modin/pandas/base.py in values(self)
   3401             The numpy representation of this object.
   3402         """
-> 3403         return self.to_numpy()
   3404 
   3405     @property

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/modin/pandas/base.py in to_numpy(self, dtype, copy)
   2999             A numpy array.
   3000         """
-> 3001         arr = self._query_compiler.to_numpy()
   3002         if dtype is not None:
   3003             return np.asarray(arr, dtype)

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/modin/backends/pandas/query_compiler.py in to_numpy(self)
    146             NumPy Array of the QueryCompiler.
    147         """
--> 148         arr = self._modin_frame.to_numpy()
    149         ErrorMessage.catch_bugs_and_request_email(
    150             len(arr) != len(self.index) or len(arr[0]) != len(self.columns)

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/modin/engines/base/frame/data.py in to_numpy(self)
   1201             Numpy array.
   1202         """
-> 1203         return self._frame_mgr_cls.to_numpy(self._partitions)
   1204 
   1205     def transpose(self):

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/modin/engines/ray/generic/frame/partition_manager.py in to_numpy(cls, partitions)
     21             [
     22                 obj.apply(lambda df: df.to_numpy()).oid
---> 23                 for row in partitions
     24                 for obj in row
     25             ]

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/modin/engines/ray/generic/frame/partition_manager.py in <listcomp>(.0)
     22                 obj.apply(lambda df: df.to_numpy()).oid
     23                 for row in partitions
---> 24                 for obj in row
     25             ]
     26         )

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/modin/engines/ray/pandas_on_ray/frame/partition.py in apply(self, func, **kwargs)
     50         oid = self.oid
     51         call_queue = self.call_queue + [(func, kwargs)]
---> 52         result, length, width = deploy_ray_func.remote(call_queue, oid)
     53         return PandasOnRayFramePartition(result, length, width)
     54 

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/remote_function.py in _remote_proxy(*args, **kwargs)
     76         @wraps(function)
     77         def _remote_proxy(*args, **kwargs):
---> 78             return self._remote(args=args, kwargs=kwargs)
     79 
     80         self.remote = _remote_proxy

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/remote_function.py in _remote(self, args, kwargs, num_return_vals, num_cpus, num_gpus, resources)
    153             invocation = self._decorator(invocation)
    154 
--> 155         return invocation(args, kwargs)

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/remote_function.py in invocation(args, kwargs)
    143                     args,
    144                     num_return_vals=num_return_vals,
--> 145                     resources=resources)
    146 
    147             if len(object_ids) == 1:

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py in submit_task(self, function_descriptor, args, actor_id, actor_handle_id, actor_counter, actor_creation_id, actor_creation_dummy_object_id, previous_actor_task_dummy_object_id, max_actor_reconstructions, new_actor_handles, num_return_vals, resources, placement_resources, job_id)
    678                     args_for_raylet.append(arg)
    679                 else:
--> 680                     args_for_raylet.append(put(arg))
    681 
    682             if new_actor_handles is None:

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py in put(value)
   2275                 worker.task_context.put_index,
   2276             )
-> 2277             worker.put_object(object_id, value)
   2278         worker.task_context.put_index += 1
   2279         return object_id

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py in put_object(self, object_id, value)
    396                 range(ray_constants.DEFAULT_PUT_OBJECT_RETRIES)):
    397             try:
--> 398                 self._try_store_and_register(object_id, value)
    399                 break
    400             except pyarrow.plasma.PlasmaStoreFull as plasma_exc:

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py in _try_store_and_register(self, object_id, value)
    416         """
    417         try:
--> 418             self.store_and_register(object_id, value)
    419         except pyarrow.plasma.PlasmaObjectExists:
    420             # The object already exists in the object store, so there is no

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/worker.py in store_and_register(self, object_id, value, depth)
    325                         memcopy_threads=self.memcopy_threads,
    326                         serialization_context=self.get_serialization_context(
--> 327                             self.current_job_id))
    328                 break
    329             except pyarrow.SerializationCallbackError as e:

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/utils.py in _wrapper(*args, **kwargs)
    517                 def _wrapper(*args, **kwargs):
    518                     with self.lock:
--> 519                         return orig_attr(*args, **kwargs)
    520 
    521                 self._wrapper_cache[attr] = _wrapper

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/pyarrow_files/pyarrow/_plasma.pyx in pyarrow._plasma.PlasmaClient.put()

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/pyarrow_files/pyarrow/_plasma.pyx in pyarrow._plasma.PlasmaClient.create()

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/pyarrow_files/pyarrow/_plasma.pyx in pyarrow._plasma.plasma_check_status()

~/Store/Cardiomics/python_dir/python_project/lib/python3.5/site-packages/ray/pyarrow_files/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowIOError: Encountered unexpected EOF

(pid=41697) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fb2d94ea730>
(pid=41697) Traceback (most recent call last):
(pid=41697)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41697) TypeError: 'NoneType' object is not callable
(pid=41697) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fb2d94ea730>
(pid=41697) Traceback (most recent call last):
(pid=41697)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41697) TypeError: 'NoneType' object is not callable
(pid=41720) Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f17b1829730>
(pid=41720) Traceback (most recent call last):
(pid=41720)   File "/BiO/Access/yahoyoungho/Store/Cardiomics/python_dir/python_project/lib/python3.5/weakref.py", line 117, in remove
(pid=41720) TypeError: 'NoneType' object is not callable

similar error for mat_np = mat.to_numpy()

@devin-petersohn
Copy link
Collaborator

Hi @yahoyoungho, thanks for posting!

This looks like an issue related to Ray. I have created an issue in the Ray repo (ray-project/ray#6198) to track this. Thanks again for reporting!

@devin-petersohn
Copy link
Collaborator

Closing this since it was fixed in Ray. Please feel free to reopen if necessary!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Ray ⚡ Issues related to the Ray engine
Projects
None yet
Development

No branches or pull requests

2 participants