[core] Make ReleaseUnusedBundles Fault Tolerant #57786

Sparks0219 · 2025-10-16T05:56:28Z

Description

Making ReleaseUnusedBundles fault tolerant and enabling retries on network failures. Added cpp test to verify idempotency and created a python integration test. Also created a fake worker class since I needed to noop the underlying connection which is used in DestroyWorker and didn't want to modify the mock class.

Related issues

Types of change

Checklist

Does this PR introduce breaking changes?

Yes ⚠️
No

Testing:

Added/updated tests for my changes
Tested the changes manually
This PR is not tested ❌ (please explain why)

Code Quality:

Signed off every commit (git commit -s)
Ran pre-commit hooks (setup guide)

Documentation:

Updated documentation (if applicable) (contribution guide)
Added new APIs to doc/source/ (if applicable)

Additional context

Signed-off-by: joshlee <joshlee@anyscale.com>

edoakes · 2025-10-16T14:19:39Z

@Sparks0219 merge conflict, and could you please separate the worker interface refactoring into another PR?

Signed-off-by: joshlee <joshlee@anyscale.com>

dayshah · 2025-10-16T21:21:20Z

src/ray/raylet/tests/node_manager_test.cc


 TEST_F(NodeManagerTest, TestHandleRequestWorkerLeaseInfeasibleIdempotent) {
-  auto lease_spec = BuildLeaseSpec({{"CPU", 1}});
+  auto lease_spec = BuildLeaseSpec({{"CPU", 11}});


why this change?

ah I needed to expand the cpu resources for this test from 0 to some positive number (10) for the bundle test, but then the infeasible lease test failed since 1 cpu is no longer infeasible. I'll change this to a constexpr and not some magic number so it's more clear

dayshah · 2025-10-16T21:22:33Z

src/ray/raylet/placement_group_resource_manager.h

      bundle_spec_map_;
+
+  friend bool IsBundleRegistered(const PlacementGroupResourceManager &manager,
+                                 const BundleID &bundle_id);


👀
no way around? + is it necessary to assert on this

mmm I think we need some way of looking into pgmanager state since ReleaseUnusedBundles is calling pgmanager methods and I don't think there's anything I can use for this in the class currently :(

dayshah · 2025-10-16T21:25:25Z

src/ray/raylet/tests/node_manager_test.cc

        *local_lease_manager_);

+    placement_group_resource_manager_ =
+        std::make_unique<NewPlacementGroupResourceManager>(*cluster_resource_scheduler_);


can save for the refactor pr, but why is called new 😵‍💫

👀 no clue

dayshah · 2025-10-16T21:54:51Z

python/ray/tests/test_raylet_fault_tolerance.py

+    monkeypatch.setenv(
+        "RAY_testing_rpc_failure",
+        "NodeManagerService.grpc_client.ReleaseUnusedBundles=1:100:0"
+        + ",NodeManagerService.grpc_client.CancelResourceReserve=100:100:0",


just do -1 if you want to kill it

but I don't love this

So like it's possible to happen without injecting failures on CancelResourceReserve because the gcs could be die before all the Cancel's happen. You should leave a comment describing that otherwise it's p hard to figure out

Yea exactly, but its inherently flaky so we need to perma block CancelResourceReserve to make it deterministic. I'll leave a comment + put -1 instead to be mroe clear about my intentions

Signed-off-by: joshlee <joshlee@anyscale.com>

dayshah

couple questions but lgtm

dayshah · 2025-10-19T02:23:44Z

python/ray/tests/test_raylet_fault_tolerance.py

+def test_release_unused_bundles_idempotent(
+    inject_rpc_failures, ray_start_cluster_head_with_external_redis
+):
+    # NOTE: Not testing response failure because the leaked bundle is cleaned up anyway


Is there any harm in testing resp failure too just for completeness?

Eh not really I can add it in, though only request will fail without this change and response passes.

dayshah · 2025-10-19T02:26:11Z

src/ray/raylet/fake_worker.h

+        port_(port),
+        proc_(Process::CreateNewDummy()),
+        connection_([&io_context]() {
+          local_stream_socket socket(io_context);


are we creating a real socket in a fake?

Yup I am, seems a bit of a pain to rework it not to take in a real socket so just passed real socket + dummy lambdas as a compromise

dayshah · 2025-10-19T02:32:49Z

src/ray/raylet/tests/node_manager_test.cc

+  // bundle_in_use == true: a bundle is marked as in use in the placement group resource
+  // manager. ReleaseUnusedBundles is expected to not release the bundle.
+  // bundle_in_use == false: a bundle is not marked as in use in the placement group
+  // resource manager. ReleaseUnusedBundles is expected to release the bundle.


bundle_in_use is state right, so there's the case where on the 1st cancel, it was in use, and on the retry it wasn't in use. But ya the behavior is correct anyways to take state into account but just not perfectly "idempotent"

True good point, any state change in the middle between the retries will cause issues with idempotency. I'll rename the test to retries instead

Signed-off-by: joshlee <joshlee@anyscale.com>

python/ray/tests/test_raylet_fault_tolerance.py

Signed-off-by: joshlee <joshlee@anyscale.com>

Signed-off-by: joshlee <joshlee@anyscale.com> Signed-off-by: Kamil Kaczmarek <kamil@anyscale.com>

Signed-off-by: joshlee <joshlee@anyscale.com> Signed-off-by: xgui <xgui@anyscale.com>

Signed-off-by: joshlee <joshlee@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Signed-off-by: joshlee <joshlee@anyscale.com>

Signed-off-by: joshlee <joshlee@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

Sparks0219 added 3 commits October 16, 2025 05:55

Make ReleaseUnusedBundles Fault Tolerant

1f9b0e4

Signed-off-by: joshlee <joshlee@anyscale.com>

nits

ba5f6b4

Signed-off-by: joshlee <joshlee@anyscale.com>

nits

a2d53b2

Signed-off-by: joshlee <joshlee@anyscale.com>

Sparks0219 requested review from dayshah and edoakes October 16, 2025 06:14

Sparks0219 added the go add ONLY when ready to merge, run all tests label Oct 16, 2025

Sparks0219 marked this pull request as ready for review October 16, 2025 06:15

Sparks0219 requested a review from a team as a code owner October 16, 2025 06:15

Merge conflicts

240ef63

Signed-off-by: joshlee <joshlee@anyscale.com>

ray-gardener bot added the core Issues that should be addressed in Ray Core label Oct 16, 2025

dayshah assigned MengjinYan Oct 16, 2025

Sparks0219 force-pushed the joshlee/make-release-unused-bundles-fault-tolerant branch from 992e1bb to 240ef63 Compare October 16, 2025 18:54

This comment was marked as outdated.

Sign in to view

Sparks0219 force-pushed the joshlee/make-release-unused-bundles-fault-tolerant branch 2 times, most recently from f896cd6 to 240ef63 Compare October 16, 2025 19:55

Sparks0219 added 2 commits October 16, 2025 19:56

nit

315511d

Signed-off-by: joshlee <joshlee@anyscale.com>

Merge conflicts

659d537

Signed-off-by: joshlee <joshlee@anyscale.com>

dayshah reviewed Oct 16, 2025

View reviewed changes

Addressing comments

892e527

Signed-off-by: joshlee <joshlee@anyscale.com>

Sparks0219 requested a review from dayshah October 17, 2025 23:23

dayshah approved these changes Oct 19, 2025

View reviewed changes

Sparks0219 added 2 commits October 20, 2025 17:08

Merge conflicts

5abf9dc

Signed-off-by: joshlee <joshlee@anyscale.com>

Addressing comments

3dc7333

Signed-off-by: joshlee <joshlee@anyscale.com>

dayshah reviewed Oct 20, 2025

View reviewed changes

python/ray/tests/test_raylet_fault_tolerance.py Outdated Show resolved Hide resolved

Sparks0219 added 4 commits October 20, 2025 19:46

Merge conflicts

7cc3e8c

Signed-off-by: joshlee <joshlee@anyscale.com>

Merge conflicts

0a9ef1d

Signed-off-by: joshlee <joshlee@anyscale.com>

Remove timeout

d3c717d

Signed-off-by: joshlee <joshlee@anyscale.com>

nits

c6e8ad9

Signed-off-by: joshlee <joshlee@anyscale.com>

dayshah approved these changes Oct 20, 2025

View reviewed changes

dayshah enabled auto-merge (squash) October 20, 2025 20:11

dayshah merged commit a9065a3 into ray-project:master Oct 20, 2025
7 checks passed

kamil-kaczmarek pushed a commit that referenced this pull request Oct 20, 2025

[core] Make ReleaseUnusedBundles Fault Tolerant (#57786)

c6ba247

Signed-off-by: joshlee <joshlee@anyscale.com> Signed-off-by: Kamil Kaczmarek <kamil@anyscale.com>

xinyuangui2 pushed a commit to xinyuangui2/ray that referenced this pull request Oct 22, 2025

[core] Make ReleaseUnusedBundles Fault Tolerant (ray-project#57786)

8b47af6

Signed-off-by: joshlee <joshlee@anyscale.com> Signed-off-by: xgui <xgui@anyscale.com>

elliot-barn pushed a commit that referenced this pull request Oct 23, 2025

[core] Make ReleaseUnusedBundles Fault Tolerant (#57786)

61e26e3

Signed-off-by: joshlee <joshlee@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

landscapepainter pushed a commit to landscapepainter/ray that referenced this pull request Nov 17, 2025

[core] Make ReleaseUnusedBundles Fault Tolerant (ray-project#57786)

0784e4a

Signed-off-by: joshlee <joshlee@anyscale.com>

Aydin-ab pushed a commit to Aydin-ab/ray-aydin that referenced this pull request Nov 19, 2025

[core] Make ReleaseUnusedBundles Fault Tolerant (ray-project#57786)

a7210f8

Signed-off-by: joshlee <joshlee@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

[core] Make ReleaseUnusedBundles Fault Tolerant #57786

[core] Make ReleaseUnusedBundles Fault Tolerant #57786

Uh oh!

Conversation

Sparks0219 commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

Types of change

Checklist

Additional context

Uh oh!

edoakes commented Oct 16, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sparks0219 Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dayshah left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Sparks0219 commented Oct 16, 2025 •

edited

Loading

Sparks0219 Oct 16, 2025 •

edited

Loading