Skip to content

Commit

Permalink
[core][cgraph] Fix doc test cgraph_overlap (#51139)
Browse files Browse the repository at this point in the history
Explicitly tear down the compiled graph and kill the actors rather than relying on GC.

Also previously in Compiled Graph, actor is killed only when the actor task does not finish within timeout. This PR fixes it by always killing the actor when kill_actors=True.

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
  • Loading branch information
ruisearch42 authored Mar 7, 2025
1 parent 0cf2a4b commit d0e76d6
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 0 deletions.
1 change: 1 addition & 0 deletions doc/source/ray-core/doc_code/cgraph_overlap.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ def test(overlap_gpu_communication):
assert result == [(i, shape, dtype)] * num_senders
duration = time.monotonic() - start
print(f"{overlap_gpu_communication=}, {duration=}")
compiled_dag.teardown(kill_actors=True)


for overlap_gpu_communication in [False, True]:
Expand Down
7 changes: 7 additions & 0 deletions python/ray/dag/compiled_dag_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -2144,6 +2144,13 @@ def wait_teardown(self, kill_actors: bool = False):
except Exception:
pass

if kill_actors:
# In the previous loop, we allow the actor tasks to exit first.
# Now, we force kill the actors if not yet.
for actor in outer.worker_task_refs:
logger.info(f"Killing actor: {actor}")
ray.kill(actor)

def teardown(self, kill_actors: bool = False):
with self._in_teardown_lock:
if self._teardown_done:
Expand Down

0 comments on commit d0e76d6

Please sign in to comment.