Skip to content

Commit 18593ab

Browse files
Future-Outlierrueian
authored andcommitted
[Doc][KubeRay] Add ReconcileConcurrency configuration instructions to Troubleshooting Guide (#55236)
## Why are these changes needed? Some users may not know how to configure the `ReconcileConcurrency` in kuberay. Docs link: https://anyscale-ray--55236.com.readthedocs.build/en/55236/cluster/kubernetes/troubleshooting/troubleshooting.html#how-to-configure-reconcile-concurrency-when-there-are-large-mount-of-crs ray-project/kuberay#3909 --------- Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
1 parent dfc6600 commit 18593ab

File tree

1 file changed

+29
-0
lines changed

1 file changed

+29
-0
lines changed

doc/source/cluster/kubernetes/troubleshooting/troubleshooting.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,3 +109,32 @@ For comprehensive troubleshooting of multi-node GPU serving issues, refer to {re
109109
### Why are changes to the RayCluster or RayJob CR not taking effect?
110110

111111
Currently, only modifications to the `replicas` field in `RayCluster/RayJob` CR are supported. Changes to other fields may not take effect or could lead to unexpected results.
112+
113+
### How to configure reconcile concurrency when there are large mount of CRs?
114+
115+
In this example, [kuberay#3909](https://github.com/ray-project/kuberay/issues/3909),
116+
the user encountered high latency when processing RayCluster CRs and found that the ReconcileConcurrency value was set to 1.
117+
118+
The KubeRay operator supports configuring the `ReconcileConcurrency` setting, which controls the number of concurrent workers processing Ray custom resources (CRs).
119+
120+
To configure the `ReconcileConcurrency` number, you can edit the deployment's container args:
121+
122+
```bash
123+
kubectl edit deployment kuberay-operator
124+
```
125+
126+
Specify the `ReconcileConcurrency` number in the container args:
127+
128+
```yaml
129+
spec:
130+
containers:
131+
- args:
132+
- --reconcile-concurrency
133+
- "10"
134+
```
135+
136+
You can also use the following command for kuberay version >= 1.5.0:
137+
138+
```bash
139+
helm install kuberay-operator kuberay/kuberay-operator --version 1.5.0 --set reconcileConcurrency=10
140+
```

0 commit comments

Comments
 (0)