[core] Adding option for in flight rpc failure injection #58512

dayshah · 2025-11-10T21:58:30Z

Description

Our current RPC failure injection framework accounts for 2 types of failures.

The callback is called Status::Unavailable and the request is never sent.
The callback is called with Status::Unavailable when the reply comes back.

It doesn't account for the case that
3. The request makes it to the server, and the callback is called with Status::Unavailable while the server is still processing the request. In these situations the client could retry and the server could get the retry while still processing the old request.

This adds an option to inject failures for that third case. In the more realistic iptable chaos tests, we found that this does happen.
This will help deterministically find issues like the one fixed here #58265

Signed-off-by: dayshah <dhyey2019@gmail.com>

cursor

Bug: Legacy Config Breaks Modern Framework.

The RPC failure configuration uses the old 3-parameter format 1:100:0 but the updated framework requires 4 parameters minimum. This will cause a RAY_CHECK_GE assertion failure when the configuration is parsed, crashing the test.

python/ray/tests/test_raylet_fault_tolerance.py#L63-L64

ray/python/ray/tests/test_raylet_fault_tolerance.py

Lines 63 to 64 in 80ff33b

    
           "RAY_testing_rpc_failure", 
        
           "NodeManagerService.grpc_client.DrainRaylet=1:100:0",

src/ray/rpc/rpc_chaos.cc

python/ray/tests/test_raylet_fault_tolerance.py

Signed-off-by: dayshah <dhyey2019@gmail.com>

Sparks0219 · 2025-11-13T23:15:39Z

python/ray/tests/test_core_worker_fault_tolerance.py



-@pytest.mark.parametrize("deterministic_failure", ["request", "response"])
+@pytest.mark.parametrize("deterministic_failure", ["request", "response", "in_flight"])


nit: use RPC_FAILURE_TYPES

Seems this comment is not addressed

Sparks0219

Could you also update the comment for TODO(#58246) in data_release_tests.yaml

jjyao

nice!

jjyao · 2025-11-13T23:35:50Z

python/ray/dashboard/modules/job/tests/test_job_manager.py

            "cmd": "ray start --head",
            "env": {
-                "RAY_testing_rpc_failure": "ray::rpc::InternalKVGcsService.grpc_client.InternalKVGet=2:50:50,CoreWorkerService.grpc_client.PushTask=3:50:50"
+                "RAY_testing_rpc_failure": "ray::rpc::InternalKVGcsService.grpc_client.InternalKVGet=3:33:33:33,CoreWorkerService.grpc_client.PushTask=3:33:33:33"


Gonna change it to a proper json

jjyao · 2025-11-13T23:37:16Z

python/ray/_private/test_utils.py

+RPC_FAILURE_MAP = {
+    "request": "100:0:0",
+    "response": "0:100:0",
+    "in_flight": "0:0:100",


seems our codebase uses it as one word?

it's supposed to be 2 lol https://www.merriam-webster.com/dictionary/in-flight

my boy cited merriam webster 💪

definitely two words: "the requests are in flight" and "the in-flight requests"

Sparks0219 · 2025-11-13T23:42:40Z

src/ray/common/ray_config_def.h

 //  it will apply to all methods.
 RAY_CONFIG(std::string, testing_asio_delay_us, "")

 /// To use this, simply do


"simply do" lol

took it out since it's not simple anymore 💀

Sparks0219 · 2025-11-13T23:44:46Z

src/ray/rpc/rpc_chaos.cc

+
+// You can also provide 5th, 6th, and / or 7th  optional parameters to specify that there
+// should be at least a certain amount of request, response, and in flight failures.
+// flight failures. By default these are set to 0, but by setting them to positive values


typo "flight failures" is twice written

Sparks0219 · 2025-11-13T23:46:38Z

src/ray/rpc/rpc_chaos.cc

+// Key is the RPC call name and value is a four part colon separated structure. It
 // contains the max number of failures to inject + probability of req failure +
-// probability of reply failure.
+// probability of reply failure + probability of in flight failure.


super nit: should be in-flight since it's an adjective

changed, grammar 💪🏽

Sparks0219 · 2025-11-13T23:56:36Z

src/ray/rpc/rpc_chaos.cc

      failable.num_remaining_failures--;
      return RpcFailure::Response;
    }
+    if (random_number <= failable.req_failure_prob + failable.resp_failure_prob +


nit: could just be <= 100.

or this could just be structured an if/else if/else

restructured to if else if else, but why <=100? all 3 numbers added up could still be <100, like 10:10:10?

Signed-off-by: dayshah <dhyey2019@gmail.com>

jjyao · 2025-11-16T23:35:08Z

python/ray/tests/test_core_worker_fault_tolerance.py



-@pytest.mark.parametrize("deterministic_failure", ["request", "response"])
+@pytest.mark.parametrize("deterministic_failure", ["request", "response", "in_flight"])


Seems this comment is not addressed

Signed-off-by: dayshah <dhyey2019@gmail.com>

…#58512) Signed-off-by: dayshah <dhyey2019@gmail.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…#58512) Signed-off-by: dayshah <dhyey2019@gmail.com> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>

…#58512) Signed-off-by: dayshah <dhyey2019@gmail.com>

[core] Adding option for in flight rpc failure injection

fe6d820

Signed-off-by: dayshah <dhyey2019@gmail.com>

dayshah added the go add ONLY when ready to merge, run all tests label Nov 10, 2025

dayshah added 2 commits November 10, 2025 15:28

common test util

b8a6f8f

Signed-off-by: dayshah <dhyey2019@gmail.com>

in _private test utils

80ff33b

Signed-off-by: dayshah <dhyey2019@gmail.com>

dayshah marked this pull request as ready for review November 10, 2025 23:32

dayshah requested a review from a team as a code owner November 10, 2025 23:32

dayshah assigned Sparks0219 Nov 10, 2025

cursor bot reviewed Nov 10, 2025

View reviewed changes

src/ray/rpc/rpc_chaos.cc Show resolved Hide resolved

python/ray/tests/test_raylet_fault_tolerance.py Show resolved Hide resolved

ray-gardener bot added the core Issues that should be addressed in Ray Core label Nov 11, 2025

fix

a2ae76f

Signed-off-by: dayshah <dhyey2019@gmail.com>

Sparks0219 reviewed Nov 13, 2025

View reviewed changes

jjyao reviewed Nov 13, 2025

View reviewed changes

Sparks0219 reviewed Nov 13, 2025

View reviewed changes

dayshah added 2 commits November 15, 2025 19:26

Merge branch 'master' into in-flight-fail

2fc1475

Signed-off-by: dayshah <dhyey2019@gmail.com>

grammar

1042226

Signed-off-by: dayshah <dhyey2019@gmail.com>

dayshah requested review from Sparks0219 and jjyao November 16, 2025 03:32

jjyao approved these changes Nov 16, 2025

View reviewed changes

comment

4b3d476

Signed-off-by: dayshah <dhyey2019@gmail.com>

dayshah mentioned this pull request Nov 17, 2025

[core] Make RPC Chaos Configurations More Readable (JSON) #58686

Open

dayshah enabled auto-merge (squash) November 17, 2025 00:56

Merge branch 'master' into in-flight-fail

e3bbebf

github-actions bot disabled auto-merge November 17, 2025 06:08

dayshah enabled auto-merge (squash) November 17, 2025 06:11

dayshah merged commit fbf3c32 into ray-project:master Nov 17, 2025
7 checks passed

dayshah deleted the in-flight-fail branch November 17, 2025 08:46

Aydin-ab pushed a commit to Aydin-ab/ray-aydin that referenced this pull request Nov 19, 2025

[core] Adding option for in flight rpc failure injection (ray-project…

cc15948

…#58512) Signed-off-by: dayshah <dhyey2019@gmail.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

dancingactor mentioned this pull request Nov 21, 2025

[core] Make RPC Chaos Configurations More Readable as JSON #58886

Open

ykdojo pushed a commit to ykdojo/ray that referenced this pull request Nov 27, 2025

[core] Adding option for in flight rpc failure injection (ray-project…

6ebaf18

…#58512) Signed-off-by: dayshah <dhyey2019@gmail.com> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>

SheldonTsen pushed a commit to SheldonTsen/ray that referenced this pull request Dec 1, 2025

[core] Adding option for in flight rpc failure injection (ray-project…

2e0e529

…#58512) Signed-off-by: dayshah <dhyey2019@gmail.com>

	"RAY_testing_rpc_failure",
	"NodeManagerService.grpc_client.DrainRaylet=1:100:0",



		@pytest.mark.parametrize("deterministic_failure", ["request", "response"])
		@pytest.mark.parametrize("deterministic_failure", ["request", "response", "in_flight"])

[core] Adding option for in flight rpc failure injection #58512

[core] Adding option for in flight rpc failure injection #58512

Uh oh!

Conversation

dayshah commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Legacy Config Breaks Modern Framework.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sparks0219 left a comment

Choose a reason for hiding this comment

Uh oh!

jjyao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edoakes Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dayshah commented Nov 10, 2025 •

edited

Loading

edoakes Nov 14, 2025 •

edited

Loading