[4/n] [Serve] Add Node Rank and Local Rank Support to Ray Serve Replica Ranks #58479

abrarsheikh · 2025-11-09T03:28:05Z

Summary

This PR extends Ray Serve's replica rank system to track node rank and local rank in addition to the existing global rank, enabling better distributed serving coordination and multi-node deployment awareness.

Changes

Core Implementation (deployment_state.py)

Extended DeploymentRankManager to maintain three levels of rank tracking:
- Global rank: Replica-level rank across all nodes (0 to N-1)
- Node rank: Index assigned to each node (0 to M-1)
- Local rank: Replica's rank within its node (0 to K-1 per node)
Modified assign_rank() to return ReplicaRank objects containing all three rank types
Added node rank manager and per-node local rank managers to track replica placement
Updated get_replica_ranks_mapping() to return Dict[str, ReplicaRank] instead of Dict[str, int]

Backward Compatibility

All existing functionality is preserved. The global rank behavior remains unchanged, with node and local ranks added as additional fields in the ReplicaRank object.

Signed-off-by: abrar <abrar@anyscale.com>

**Summary** Modified replica rank assignment to defer rank allocation until the replica is actually allocated, rather than assigning it during the startup call. This is necessary when we want to add node local rank in future, in order to support node rank and node local rank we need to know the node_id which is only known after replica is allocated. **Changes** - Changed `start()` method signature to accept `assign_rank_callback` instead of a pre-assigned `rank` parameter - Rank is now assigned after `_allocated_obj_ref` is resolved, ensuring the replica is allocated before rank assignment - Pass rank to `initialize_and_get_metadata()` method on the replica actor, allowing rank to be set during initialization - Updated `ReplicaBase.initialize()` to accept rank as a parameter and set it along with the internal replica context - Added `PENDING_INITIALIZATION` status check to handle cases where `_ready_obj_ref` is not yet set Next PR #58479 --------- Signed-off-by: abrar <abrar@anyscale.com>

…ar-rank-p4

Signed-off-by: abrar <abrar@anyscale.com>

…roject#58477) **Summary** Modified replica rank assignment to defer rank allocation until the replica is actually allocated, rather than assigning it during the startup call. This is necessary when we want to add node local rank in future, in order to support node rank and node local rank we need to know the node_id which is only known after replica is allocated. **Changes** - Changed `start()` method signature to accept `assign_rank_callback` instead of a pre-assigned `rank` parameter - Rank is now assigned after `_allocated_obj_ref` is resolved, ensuring the replica is allocated before rank assignment - Pass rank to `initialize_and_get_metadata()` method on the replica actor, allowing rank to be set during initialization - Updated `ReplicaBase.initialize()` to accept rank as a parameter and set it along with the internal replica context - Added `PENDING_INITIALIZATION` status check to handle cases where `_ready_obj_ref` is not yet set Next PR ray-project#58479 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

Signed-off-by: abrar <abrar@anyscale.com>

python/ray/serve/_private/deployment_state.py

Signed-off-by: abrar <abrar@anyscale.com>

…roject#58477) **Summary** Modified replica rank assignment to defer rank allocation until the replica is actually allocated, rather than assigning it during the startup call. This is necessary when we want to add node local rank in future, in order to support node rank and node local rank we need to know the node_id which is only known after replica is allocated. **Changes** - Changed `start()` method signature to accept `assign_rank_callback` instead of a pre-assigned `rank` parameter - Rank is now assigned after `_allocated_obj_ref` is resolved, ensuring the replica is allocated before rank assignment - Pass rank to `initialize_and_get_metadata()` method on the replica actor, allowing rank to be set during initialization - Updated `ReplicaBase.initialize()` to accept rank as a parameter and set it along with the internal replica context - Added `PENDING_INITIALIZATION` status check to handle cases where `_ready_obj_ref` is not yet set Next PR ray-project#58479 --------- Signed-off-by: abrar <abrar@anyscale.com>

…roject#58477) **Summary** Modified replica rank assignment to defer rank allocation until the replica is actually allocated, rather than assigning it during the startup call. This is necessary when we want to add node local rank in future, in order to support node rank and node local rank we need to know the node_id which is only known after replica is allocated. **Changes** - Changed `start()` method signature to accept `assign_rank_callback` instead of a pre-assigned `rank` parameter - Rank is now assigned after `_allocated_obj_ref` is resolved, ensuring the replica is allocated before rank assignment - Pass rank to `initialize_and_get_metadata()` method on the replica actor, allowing rank to be set during initialization - Updated `ReplicaBase.initialize()` to accept rank as a parameter and set it along with the internal replica context - Added `PENDING_INITIALIZATION` status check to handle cases where `_ready_obj_ref` is not yet set Next PR ray-project#58479 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>

…roject#58477) **Summary** Modified replica rank assignment to defer rank allocation until the replica is actually allocated, rather than assigning it during the startup call. This is necessary when we want to add node local rank in future, in order to support node rank and node local rank we need to know the node_id which is only known after replica is allocated. **Changes** - Changed `start()` method signature to accept `assign_rank_callback` instead of a pre-assigned `rank` parameter - Rank is now assigned after `_allocated_obj_ref` is resolved, ensuring the replica is allocated before rank assignment - Pass rank to `initialize_and_get_metadata()` method on the replica actor, allowing rank to be set during initialization - Updated `ReplicaBase.initialize()` to accept rank as a parameter and set it along with the internal replica context - Added `PENDING_INITIALIZATION` status check to handle cases where `_ready_obj_ref` is not yet set Next PR ray-project#58479 --------- Signed-off-by: abrar <abrar@anyscale.com>

eicherseiji

LGTM. Thanks @abrarsheikh!

python/ray/serve/_private/deployment_state.py

eicherseiji · 2025-12-02T23:00:43Z

python/ray/serve/_private/deployment_state.py


        return self._execute_with_error_handling(
-            _get_replica_rank_impl, ReplicaRank(rank=0, node_rank=-1, local_rank=-1)
+            _get_replica_rank_impl, ReplicaRank(rank=0, node_rank=0, local_rank=0)


For my information, why change the default from -1 -> 0?

-1 was a placeholder, for a previous stacked diff.

…ar-rank-p4

Signed-off-by: abrar <abrar@anyscale.com>

cursor · 2025-12-03T00:43:25Z

python/ray/serve/_private/deployment_state.py

+            self._local_rank_managers[node_id].recover_rank(replica_id, rank.local_rank)
+
+            # Track the replica-to-node mapping
+            self._replica_to_node[replica_id] = node_id


Bug: Inconsistent state on partial failure in recover_rank

The order of operations in _recover_rank_impl is inconsistent with _assign_rank_impl. In _assign_rank_impl, _replica_to_node[replica_id] = node_id is set first (before any rank assignments), but in _recover_rank_impl, it's set last (after all rank recoveries). When _fail_on_rank_error=False and an error occurs after recovering the global rank but before setting _replica_to_node, the system ends up in an inconsistent state where _replica_rank_manager has the replica's global rank but _replica_to_node doesn't have the mapping. This causes has_replica_rank() to return False even though ranks are partially assigned, potentially leading to duplicate assignment errors on retry.

Additional Locations (1)

python/ray/serve/_private/deployment_state.py#L1717-L1725

abrarsheikh added 7 commits November 8, 2025 02:18

Refactor replica rank to prepare for node local ranks

8e8f393

Signed-off-by: abrar <abrar@anyscale.com>

referance schema

5f118c2

Signed-off-by: abrar <abrar@anyscale.com>

[Serve] Refactor replica rank to prepare for node local ranks

0f7ae3c

Signed-off-by: abrar <abrar@anyscale.com>

pass rank into replicas initialize method

8d4fbbc

Signed-off-by: abrar <abrar@anyscale.com>

fix test

51f2935

Signed-off-by: abrar <abrar@anyscale.com>

fix java test

d66a7b5

Signed-off-by: abrar <abrar@anyscale.com>

Add Node Rank and Local Rank Support to Ray Serve Replica Ranks

438b7a6

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh added the go add ONLY when ready to merge, run all tests label Nov 9, 2025

abrarsheikh mentioned this pull request Nov 9, 2025

[3/n] [Serve] Defer rank assignment after replica is allocated #58477

Merged

check rank before releasing

13a7590

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh changed the title ~~Add Node Rank and Local Rank Support to Ray Serve Replica Ranks~~ [4/n] [Serve] Add Node Rank and Local Rank Support to Ray Serve Replica Ranks Nov 9, 2025

Base automatically changed from LLM-2497-abrar-rank-p3 to master November 19, 2025 18:04

abrarsheikh added 2 commits November 19, 2025 18:36

Merge branch 'master' of github.com:ray-project/ray into LLM-2497-abr…

bcfd10e

…ar-rank-p4

fix test

e680ea0

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh added 3 commits November 19, 2025 19:22

test

e98b6b1

Signed-off-by: abrar <abrar@anyscale.com>

test

48b9426

Signed-off-by: abrar <abrar@anyscale.com>

restore build files

f4f696f

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh marked this pull request as ready for review November 19, 2025 19:32

abrarsheikh requested a review from a team as a code owner November 19, 2025 19:32

abrarsheikh requested review from akyang-anyscale and kouroshHakha November 19, 2025 19:32

cursor bot reviewed Nov 19, 2025

View reviewed changes

python/ray/serve/_private/deployment_state.py Show resolved Hide resolved

ray-gardener bot added the serve Ray Serve Related Issue label Nov 20, 2025

move code

8d5077f

Signed-off-by: abrar <abrar@anyscale.com>

akyang-anyscale approved these changes Nov 24, 2025

View reviewed changes

eicherseiji approved these changes Dec 2, 2025

View reviewed changes

abrarsheikh added 2 commits December 3, 2025 00:35

Merge branch 'master' of github.com:ray-project/ray into LLM-2497-abr…

79a60be

…ar-rank-p4

update doc

1a494aa

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh enabled auto-merge (squash) December 3, 2025 00:36

cursor bot reviewed Dec 3, 2025

View reviewed changes

abrarsheikh merged commit f232a80 into master Dec 3, 2025
7 checks passed

abrarsheikh deleted the LLM-2497-abrar-rank-p4 branch December 3, 2025 02:04

eicherseiji mentioned this pull request Dec 3, 2025

[Serve] The assigned ranks for replicas through controller should be ordinal #57059

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[4/n] [Serve] Add Node Rank and Local Rank Support to Ray Serve Replica Ranks #58479

[4/n] [Serve] Add Node Rank and Local Rank Support to Ray Serve Replica Ranks #58479

Uh oh!

abrarsheikh commented Nov 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

eicherseiji left a comment

Uh oh!

Uh oh!

eicherseiji Dec 2, 2025

Uh oh!

abrarsheikh Dec 3, 2025

Uh oh!

cursor bot Dec 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[4/n] [Serve] Add Node Rank and Local Rank Support to Ray Serve Replica Ranks #58479

[4/n] [Serve] Add Node Rank and Local Rank Support to Ray Serve Replica Ranks #58479

Uh oh!

Conversation

abrarsheikh commented Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Backward Compatibility

Uh oh!

Uh oh!

eicherseiji left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eicherseiji Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

abrarsheikh Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

cursor bot Dec 3, 2025

Choose a reason for hiding this comment

Bug: Inconsistent state on partial failure in recover_rank

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abrarsheikh commented Nov 9, 2025 •

edited

Loading