[core] Actor restarts don't work when an actor creation arg is evicted from plasma

[dayshah]

What happened + What you expected to happen

Currently arguments for actor creation tasks in plasma don't have the correct lineage_ref_count. This means they can be evicted or the task specs needed to reconstruct them can be evicted. That means the max_restarts # on the actor will not be honored and we'll hit a ray.exceptions.ReferenceCountingAssertionError, when the actor tries to restart.

The solution will be to have an actor manager on the owner worker to handle actors owned by that worker. Behavior for detached actors will remain the same as it is today. Context for solution decision #51653 (comment)

Related:
#53713
#51653

Versions / Dependencies

Any Ray version

Reproduction script

cluster = Cluster()
cluster.add_node(num_cpus=0)  # head
ray.init(address=cluster.address)
worker1 = cluster.add_node(num_cpus=1)

@ray.remote(num_cpus=1, max_restarts=1)
class Actor:
    def __init__(self, config):
        self.config = config

    def ping(self):
        return self.config

# Arg is >100kb so will go in the object store
actor = Actor.remote(np.zeros(100 * 1024 * 1024, dtype=np.uint8))
ray.get(actor.ping.remote())

worker2 = cluster.add_node(num_cpus=1)
cluster.remove_node(worker1, allow_graceful=True)

# This line will break
ray.get(actor.ping.remote())

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[rclough]

After upgrading to ray 2.48 (from a few versions back) I get a whole bunch of these warnings in my logs - is there anything I can actually do about these warnings, or is it something that ray just has to fix eventually in a future version?

[dayshah]

So there's couple things you/we could do here @rclough

1. If the warning log is just printing excessively because it's per actor, but you have a workaround, we can only log per 1000 or per 1 minute or something. I think I'll just do this regardless because spamming per actor creation does seem excessive. We can also tune it down from ERROR level to WARNING level, but ideally we do want users to be aware of this hole if they have an actor that they expect to restart with init args that are stored in the object store.
2. If your argument is somewhere around ~100kb-1mb you can bump up this config / env var to make the 100kb threshold higher

   ray/src/ray/common/ray_config_def.h

   Line 195 in 6b703a8

   RAY_CONFIG(int64_t, max_direct_call_object_size, 100 * 1024)

   If you pass your arg in directly now, it should be sent inline and not go through plasma. This isn't something that's been tested much though.
3. You could also do something like ray.put the arg on the driver and never let go of the reference as long as you have the actor handle around. Basically you just need to have ray never garbage collect the object from the object store.
4. If you do need actor restarts with an actor initialization arg in plasma, that's something that we just have to fix in a later version. cc @edoakes

[a416297338]

In the reinforcement learning DDP distributed Kubernetes environment, after updating to version 2.48, I encountered an issue that causes some MultiAgentEnvRunner instances to consistently fail to create, leaving the entire environment in a waiting state and unable to run normally. Is there currently any solution to bypass this issue, allowing MultiAgentEnvRunner to automatically retry creation upon failure? Additionally, when is this issue expected to be fixed? Thank you very much!

[dayshah]

In the reinforcement learning DDP distributed Kubernetes environment, after updating to version 2.48, I encountered an issue that causes some MultiAgentEnvRunner instances to consistently fail to create, leaving the entire environment in a waiting state and unable to run normally. Is there currently any solution to bypass this issue, allowing MultiAgentEnvRunner to automatically retry creation upon failure? Additionally, when is this issue expected to be fixed? Thank you very much!

The ways to bypass this issue are the ones I listed in the comment right above. I think this is something that should be fixed this year!

[anxietymonger]

You could also do something like ray.put the arg on the driver and never let go of the reference as long as you have the actor handle around. Basically you just need to have ray never garbage collect the object from the object store.

@dayshah could you please provide an example of this method? I'm not sure if I simply call my_ref = ray.put(my_large_arg_obj), does my_ref refer to the same object in the shared object store with the one the actor uses?

[rclough]

@dayshah Just getting back around to this - funny enough, its not even clear to me exactly what I'm doing to cause the message, since I think the actors are generated for me (they are generic _MapWorker). On ~16 CPU node cluster, I get 392 of these logged over the course of about 15 seconds:

[2025-08-26 16:26:24,313 E 1781 1781] core_worker.cc:2740: Actor with class name: '_MapWorker' and ID: 'bf89cddde68a3d9702fa25aa02000000' has constructor arguments in the object store and max_restarts > 0. If the arguments in the object store go out of scope or are lost, the actor restart will fail. See https://github.com/ray-project/ray/issues/53727 for more details.

I'll have to take a closer look at what I'm mapping for what's being used in the object store, but just for more context on the lack of clarity with the error/how to fix it. I'm mapping a batch inference job, so I suspect it's something about the model... Same question as anxiety monger above, maybe I just need to explicitly put the model and then pass the ref?

[dayshah]

Sorry for the late reply @anxietymonger and @rclough

I'm not sure if I simply call my_ref = ray.put(my_large_arg_obj), does my_ref refer to the same object in the shared object store with the one the actor uses?

If you do something like

my_ref = ray.put(my_large_arg_obj)
actor = Actor.remote(my_ref)

yes the actor will use the same object in the object store as my_ref. If you keep my_ref

On ~16 CPU node cluster, I get 392 of these logged over the course of about 15 seconds:

so the log has been dropped down to once per process, so you shouldn't get spammed as much from Ray 2.49.

It seems like Ray Data is creating the _MapWorker actor, I can ask someone working on Ray Data what affects the constructor arg size + how to get it below 100kb.