|
| 1 | +(serve-replica-ranks)= |
| 2 | + |
| 3 | +# Replica ranks |
| 4 | + |
| 5 | +:::{warning} |
| 6 | +This API is experimental and may change between Ray minor versions. |
| 7 | +::: |
| 8 | + |
| 9 | +Replica ranks provide a unique identifier for **each replica within a deployment**. Each replica receives a **rank (an integer from 0 to N-1)** and **a world size (the total number of replicas)**. |
| 10 | + |
| 11 | +## Access replica ranks |
| 12 | + |
| 13 | +You can access the rank and world size from within a deployment through the replica context using [`serve.get_replica_context()`](../api/doc/ray.serve.get_replica_context.rst). |
| 14 | + |
| 15 | +The following example shows how to access replica rank information: |
| 16 | + |
| 17 | +```{literalinclude} ../doc_code/replica_rank.py |
| 18 | +:start-after: __replica_rank_start__ |
| 19 | +:end-before: __replica_rank_end__ |
| 20 | +:language: python |
| 21 | +``` |
| 22 | + |
| 23 | +```{literalinclude} ../doc_code/replica_rank.py |
| 24 | +:start-after: __replica_rank_start_run_main__ |
| 25 | +:end-before: __replica_rank_end_run_main__ |
| 26 | +:language: python |
| 27 | +``` |
| 28 | + |
| 29 | +The [`ReplicaContext`](../api/doc/ray.serve.context.ReplicaContext.rst) provides two key fields: |
| 30 | + |
| 31 | +- `rank`: An integer from 0 to N-1 representing this replica's unique identifier. |
| 32 | +- `world_size`: The target number of replicas for the deployment. |
| 33 | + |
| 34 | +## Handle rank changes with reconfigure |
| 35 | + |
| 36 | +When a replica's rank changes (such as during downscaling), Ray Serve can automatically call the `reconfigure` method on your deployment class to notify it of the new rank. This allows you to update replica-specific state when ranks change. |
| 37 | + |
| 38 | +The following example shows how to implement `reconfigure` to handle rank changes: |
| 39 | + |
| 40 | +```{literalinclude} ../doc_code/replica_rank.py |
| 41 | +:start-after: __reconfigure_rank_start__ |
| 42 | +:end-before: __reconfigure_rank_end__ |
| 43 | +:language: python |
| 44 | +``` |
| 45 | + |
| 46 | +```{literalinclude} ../doc_code/replica_rank.py |
| 47 | +:start-after: __reconfigure_rank_start_run_main__ |
| 48 | +:end-before: __reconfigure_rank_end_run_main__ |
| 49 | +:language: python |
| 50 | +``` |
| 51 | + |
| 52 | +### When reconfigure is called |
| 53 | + |
| 54 | +Ray Serve automatically calls your `reconfigure` method in the following situations: |
| 55 | + |
| 56 | +1. **At replica startup:** When a replica starts, if your deployment has both a `reconfigure` method and a `user_config`, Ray Serve calls `reconfigure` after running `__init__`. This lets you initialize rank-aware state without duplicating code between `__init__` and `reconfigure`. |
| 57 | +2. **When you update user_config:** When you redeploy with a new `user_config`, Ray Serve calls `reconfigure` on all running replicas. If your `reconfigure` method includes `rank` as a parameter, Ray Serve passes both the new `user_config` and the current rank. |
| 58 | +3. **When a replica's rank changes:** During downscaling, ranks may be reassigned to maintain contiguity (0 to N-1). If your `reconfigure` method includes `rank` as a parameter and your deployment has a `user_config`, Ray Serve calls `reconfigure` with the existing `user_config` and the new rank. |
| 59 | + |
| 60 | +:::{note} |
| 61 | +**Requirements to receive rank updates:** |
| 62 | + |
| 63 | +To get rank changes through `reconfigure`, your deployment needs: |
| 64 | +- A class-based deployment (function deployments don't support `reconfigure`) |
| 65 | +- A `reconfigure` method with `rank` as a parameter: `def reconfigure(self, user_config, rank: int)` |
| 66 | +- A `user_config` in your deployment (even if it's just an empty dict: `user_config={}`) |
| 67 | + |
| 68 | +Without a `user_config`, Ray Serve won't call `reconfigure` for rank changes. |
| 69 | +::: |
| 70 | + |
| 71 | +:::{tip} |
| 72 | +If you'd like different behavior for when `reconfigure` is called with rank changes, [open a GitHub issue](https://github.com/ray-project/ray/issues/new/choose) to discuss your use case with the Ray Serve team. |
| 73 | +::: |
| 74 | + |
| 75 | +## How replica ranks work |
| 76 | + |
| 77 | +:::{note} |
| 78 | +**Rank reassignment is eventually consistent** |
| 79 | + |
| 80 | +When replicas are removed during downscaling, rank reassignment to maintain contiguity (0 to N-1) doesn't happen immediately. The controller performs rank consistency checks and reassignment only when the deployment reaches a `HEALTHY` state in its update loop. This means there can be a brief period after downscaling where ranks are non-contiguous before the controller reassigns them. |
| 81 | + |
| 82 | +This design choice prevents rank reassignment from interfering with ongoing deployment updates and rollouts. If you need immediate rank reassignment or different behavior, [open a GitHub issue](https://github.com/ray-project/ray/issues/new/choose) to discuss your use case with the Ray Serve team. |
| 83 | +::: |
| 84 | + |
| 85 | +:::{note} |
| 86 | +**Ranks don't influence scheduling or eviction decisions** |
| 87 | + |
| 88 | +Replica ranks are independent of scheduling and eviction decisions. The deployment scheduler doesn't consider ranks when placing replicas on nodes, so there's no guarantee that replicas with contiguous ranks (such as rank 0 and rank 1) will be on the same node. Similarly, during downscaling, the autoscaler's eviction decisions don't take replica ranks into account—any replica can be chosen for removal regardless of its rank. |
| 89 | + |
| 90 | +If you need rank-aware scheduling or eviction (for example, to colocate replicas with consecutive ranks), [open a GitHub issue](https://github.com/ray-project/ray/issues/new/choose) to discuss your requirements with the Ray Serve team. |
| 91 | +::: |
| 92 | + |
| 93 | +Ray Serve manages replica ranks automatically throughout the deployment lifecycle. The system maintains these invariants: |
| 94 | + |
| 95 | +1. Ranks are contiguous integers from 0 to N-1. |
| 96 | +2. Each running replica has exactly one rank. |
| 97 | +3. No two replicas share the same rank. |
| 98 | + |
| 99 | +### Rank assignment lifecycle |
| 100 | + |
| 101 | +The following table shows how ranks and world size behave during different events: |
| 102 | + |
| 103 | +| Event | Local Rank | World Size | |
| 104 | +|-------|------------|------------| |
| 105 | +| Upscaling | No change for existing replicas | Increases to target count | |
| 106 | +| Downscaling | Can change to maintain contiguity | Decreases to target count | |
| 107 | +| Other replica dies(will be restarted) | No change | No change | |
| 108 | +| Self replica dies | No change | No change | |
| 109 | + |
| 110 | +:::{note} |
| 111 | +World size always reflects the target number of replicas configured for the deployment, not the current number of running replicas. During scaling operations, the world size updates immediately to the new target, even while replicas are still starting or stopping. |
| 112 | +::: |
| 113 | + |
| 114 | +### Rank lifecycle state machine |
| 115 | + |
| 116 | +``` |
| 117 | +┌─────────────────────────────────────────────────────────────┐ |
| 118 | +│ DEPLOYMENT LIFECYCLE │ |
| 119 | +└─────────────────────────────────────────────────────────────┘ |
| 120 | +
|
| 121 | +Initial Deployment / Upscaling: |
| 122 | +┌──────────┐ assign ┌──────────┐ |
| 123 | +│ No Rank │ ───────────────> │ Rank: N-1│ |
| 124 | +└──────────┘ └──────────┘ |
| 125 | + (Contiguous: 0, 1, 2, ..., N-1) |
| 126 | +
|
| 127 | +Replica Crash: |
| 128 | +┌──────────┐ release ┌──────────┐ assign ┌──────────┐ |
| 129 | +│ Rank: K │ ───────────────> │ Released │ ────────────> │ Rank: K │ |
| 130 | +│ (Dead) │ │ │ │ (New) │ |
| 131 | +└──────────┘ └──────────┘ └──────────┘ |
| 132 | +(K can be any rank from 0 to N-1) |
| 133 | +
|
| 134 | +:::{note} |
| 135 | +When a replica crashes, Ray Serve automatically starts a replacement replica and assigns it the **same rank** as the crashed replica. This ensures rank contiguity is maintained without reassigning other replicas. |
| 136 | +::: |
| 137 | +
|
| 138 | +Downscaling: |
| 139 | +┌──────────┐ release ┌──────────┐ |
| 140 | +│ Rank: K │ ───────────────> │ Released │ |
| 141 | +│ (Stopped)│ │ │ |
| 142 | +└──────────┘ └──────────┘ |
| 143 | + │ |
| 144 | + └──> Remaining replicas may be reassigned to maintain |
| 145 | + contiguity: [0, 1, 2, ..., M-1] where M < N |
| 146 | +(K can be any rank from 0 to N-1) |
| 147 | +
|
| 148 | +Controller Recovery: |
| 149 | +┌──────────┐ recover ┌──────────┐ |
| 150 | +│ Running │ ───────────────> │ Rank: N │ |
| 151 | +│ Replicas │ │(Restored)│ |
| 152 | +└──────────┘ └──────────┘ |
| 153 | +(Controller queries replicas to reconstruct rank state) |
| 154 | +``` |
| 155 | + |
| 156 | +### Detailed lifecycle events |
| 157 | + |
| 158 | +1. **Rank assignment on startup**: Ranks are assigned when replicas start, such as during initial deployment, cold starts, or upscaling. The controller assigns ranks and propagates them to replicas during initialization. New replicas receive the lowest available rank. |
| 159 | + |
| 160 | +2. **Rank release on shutdown**: Ranks are released only after a replica fully stops, which occurs during graceful shutdown or downscaling. Ray Serve preserves existing rank assignments as much as possible to minimize disruption. |
| 161 | + |
| 162 | +3. **Handling replica crashes**: If a replica crashes unexpectedly, the system releases its rank and assigns the **same rank** to the replacement replica. This means if replica with rank 3 crashes, the new replacement replica will also receive rank 3. The replacement receives its rank during initialization, and other replicas keep their existing ranks unchanged. |
| 163 | + |
| 164 | +4. **Controller crash and recovery**: When the controller recovers from a crash, it reconstructs the rank state by querying all running replicas for their assigned ranks. Ranks aren't checkpointed; the system re-learns them directly from replicas during recovery. |
| 165 | + |
| 166 | +5. **Maintaining rank contiguity**: After downscaling, the system may reassign ranks to remaining replicas to maintain contiguity (0 to N-1). Ray Serve minimizes reassignments by only changing ranks when necessary. |
0 commit comments