Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] ray.actor.exit_actor() does not seem to work from within an async background thread #49451

Closed
shomilj opened this issue Dec 26, 2024 · 3 comments · Fixed by #49918
Closed
Assignees
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order

Comments

@shomilj
Copy link
Contributor

shomilj commented Dec 26, 2024

What happened + What you expected to happen

In actor initialization, I want to spawn an async background routine that has the ability to call ray.actor.exit_actor(); however, that API doesn't seem to actually work -

>>> AsyncActor.remote()
2024-12-26 14:20:31,392 INFO worker.py:1743 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265 
Actor(AsyncActor, e6db65d56c5f1e1f1122d91301000000)
>>> (AsyncActor pid=2151093) Exiting actor

The actor stays alive (according to Ray dashboard) instead of exiting as I would expect.

Versions / Dependencies

Ray 2.40.0

Reproduction script

import ray
import asyncio

@ray.remote
class AsyncActor:
    def __init__(self) -> None:
        self.task = asyncio.get_running_loop().create_task(
            self.start()
        )

    async def start(self):
        print("Exiting actor")
        ray.actor.exit_actor()
        print("Exited actor")

Issue Severity

Medium: It is a significant difficulty but I can work around it.

@shomilj shomilj added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Dec 26, 2024
@jcotant1 jcotant1 added the core Issues that should be addressed in Ray Core label Dec 31, 2024
@jjyao jjyao added P0 Issues that should be fixed in short order and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jan 7, 2025
@MortalHappiness
Copy link
Member

MortalHappiness commented Jan 8, 2025

@shomilj Is this bug only occurring occasionally, or can it always be reproduced using your reproduction script? I tried your script but couldn't reproduce it.

Update: I can reproduce it now.

@MortalHappiness MortalHappiness self-assigned this Jan 8, 2025
@edoakes
Copy link
Collaborator

edoakes commented Jan 14, 2025

@MortalHappiness I think we can fix this by unifying the asyncio exit path with the check_signals codepath:

if (options_.check_signals) {

Basically, when exit_actor is called, it can set a threading.Event that is checked during check_signals (which is already called periodically by the core worker).

@MortalHappiness
Copy link
Member

@edoakes Thanks for your information!

MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Jan 17, 2025
…o terminate async actor

Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Jan 17, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Jan 17, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Jan 17, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Jan 17, 2025
…o terminate async actor

Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Jan 17, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Jan 20, 2025
…o terminate async actor

Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Jan 20, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 25, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 25, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 25, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 26, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 26, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 26, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 26, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 28, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 28, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Feb 28, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 4, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 4, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 4, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 4, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 4, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 5, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 7, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 7, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 7, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
MortalHappiness added a commit to MortalHappiness/ray that referenced this issue Mar 7, 2025
Closes: ray-project#49451
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
edoakes added a commit that referenced this issue Mar 7, 2025
When an asyncio task creates another asyncio task, raising
`AsyncioActorExit` cannot make the caller exit because they are not the
same task. Therefore, this PR makes `exit_actor` to request actor exit
in core worker context, which will be checked regularly by core worker.

Closes: #49451

---------

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
elimelt pushed a commit to elimelt/ray that referenced this issue Mar 9, 2025
…t#49918)

When an asyncio task creates another asyncio task, raising
`AsyncioActorExit` cannot make the caller exit because they are not the
same task. Therefore, this PR makes `exit_actor` to request actor exit
in core worker context, which will be checked regularly by core worker.

Closes: ray-project#49451

---------

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants