[RayJob] Lift cluster status while initializing #4191
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spencer-p
requested review from
MortalHappiness,
andrewsykim,
kevin85421 and
rueian
as code owners
Contributor
Author
|
I also noticed that an updated ray cluster status is silently discarded unless the job status has changed, which may further obscure runtime issues after job submission. IMO this is another opportunity for improvement and would be happy to contribute that as well. |
Member
|
This seems like an obvious fix to me @rueian @Future-Outlier should we try to include this for v1.5.1? |
Signed-off-by: Spencer Peterson <spencerjp@google.com>
spencer-p
force-pushed
the
job-status
branch
from
1eb2a9b to
10f3cbe
Compare
Contributor
Author
|
Rebased to use changes from #4192 which addresses my comment above :) |
rueian
approved these changes
Nov 17, 2025
Collaborator
Yes, we should. |
kevin85421
reviewed
Nov 18, 2025
| if clientURL := rayJobInstance.Status.DashboardURL; clientURL == "" { | ||
| if rayClusterInstance.Status.State != rayv1.Ready { | ||
| logger.Info("Wait for the RayCluster.Status.State to be ready before submitting the job.", "RayCluster", rayClusterInstance.Name, "State", rayClusterInstance.Status.State) | ||
| // This is one of only 3 places where we update the RayJob status. For observability |
Member
There was a problem hiding this comment.
We should try our best to avoid calling UpdateStatus in different places. This causes a lot of issues before (that's why I fully rewrite the RayJob controller).
Ideally, we should make every reconciliation go through r.updateRayJobStatus at L430 and update the status there. It is OK if this is a blocker for the patch release (but I guess no).
Please revisit it @rueian @spencer-p. Thanks.
Collaborator
There was a problem hiding this comment.
How about using goto to jump to L430?
Member
There was a problem hiding this comment.
How about using goto to jump to L430?
I am not a fan of goto (goto is discouraged generally). Maybe wrap switch into a function instead to avoid returning directly.
func newFunction(...) (ctrl.Result, error) {
switch ...
}
result, err := newFunction(...)
# update status and handle error
spencer-p
added a commit
to spencer-p/kuberay
that referenced
this pull request
Nov 18, 2025
Fast follow to ray-project#4191 Signed-off-by: Spencer Peterson <spencerjp@google.com>
4 tasks
rueian
pushed a commit
that referenced
this pull request
Nov 18, 2025
Fast follow to #4191 Signed-off-by: Spencer Peterson <spencerjp@google.com>
andrewsykim
added a commit
that referenced
this pull request
Nov 21, 2025
* [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted (#4141) * [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted Signed-off-by: 400Ping <fourhundredping@gmail.com> * [Fix] Fix e2e error Signed-off-by: 400Ping <fourhundredping@gmail.com> * [Fix] fix according to rueian's comment Signed-off-by: 400Ping <fourhundredping@gmail.com> * [Chore] fix ci error Signed-off-by: 400Ping <fourhundredping@gmail.com> * Update ray-operator/controllers/ray/raycluster_controller.go Co-authored-by: Han-Ju Chen (Future-Outlier) <eric901201@gmail.com> Signed-off-by: Ping <fourhundredping@gmail.com> * Update ray-operator/controllers/ray/rayjob_controller.go Co-authored-by: Han-Ju Chen (Future-Outlier) <eric901201@gmail.com> Signed-off-by: Ping <fourhundredping@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * Trigger CI Signed-off-by: Future-Outlier <eric901201@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com> Signed-off-by: Ping <fourhundredping@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Han-Ju Chen (Future-Outlier) <eric901201@gmail.com> * fix: dashboard build for kuberay 1.5.0 (#4161) Signed-off-by: Future-Outlier <eric901201@gmail.com> * [Feature Enhancement] Set ordered replica index label to support multi-slice (#4163) * [Feature Enhancement] Set ordered replica index label to support multi-slice Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> * rename replica-id -> replica-name Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> * Separate replica index feature gate logic Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> * remove index arg in createWorkerPod Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> --------- Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> * update stale feature gate comments (#4174) Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * [RayCluster] Add more context why we don't recreate head Pod for RayJob (#4175) Signed-off-by: Kai-Hsun Chen <khchen@x.ai> * feature: Remove empty resource list initialization. (#4168) Fixes #4142. * [Dockerfile] [KubeRay Dashboard]: Fix Dockerfile warnings (ENV format, CMD JSON args) (#4167) * [#4166] improvement: Fix Dockerfile warnings (ENV format, CMD JSON args) * extract the hostname from CMD Signed-off-by: Neo Chien <6762509+cchung100m@users.noreply.github.com> --------- Signed-off-by: Neo Chien <6762509+cchung100m@users.noreply.github.com> Co-authored-by: cchung100m <cchung100m@users.noreply.github.com> * [Fix] Resolve int32 overflow by having the calculation in int64 and c… (#4158) * [Fix] Resolve int32 overflow by having the calculation in int64 and cap it if the count is over math.MaxInt32 Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Test] Add unit tests for CalculateReadyReplicas Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Fix] Add a nosec comment to pass the Lint (pre-commit) test Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Refactor] Add CapInt64ToInt32 to replace #nosec directives Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Refactor] Rename function to SafeInt64ToInt32 and add a underflowing prevention (it also help pass the lint test) Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Refactor] Remove the early return as SafeInt64ToInt32 handles the int32 overflow and underflow checking. Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> --------- Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * Add RayService incremental upgrade sample for guide (#4164) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> * Edit RayCluster example config for label selectors (#4151) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> * [RayJob] update light weight submitter image from quay.io (#4181) Signed-off-by: Future-Outlier <eric901201@gmail.com> * [flaky] RayJob fails when head Pod is deleted when job is running (#4182) Signed-off-by: Future-Outlier <eric901201@gmail.com> * [CI] Pin Docker api version to avoid API version mismatch (#4188) Signed-off-by: win5923 <ken89@kimo.com> * Make replicas configurable for kuberay-operator #4180 (#4195) * Make replicas configurable for kuberay-operator #4180 * Make replicas configurable for kuberay-operator #4180 * [Fix] rayjob update raycluster status (#4192) * feat: check if raycluster status update in rayjob * test: e2e test to check the rayjob raycluster status update * fix: dashboard http client tests discovered and passing (#4173) Signed-off-by: alimaazamat <alima.azamat2003@gmail.com> * [RayJob] Lift cluster status while initializing (#4191) Signed-off-by: Spencer Peterson <spencerjp@google.com> * [RayJob] Remove updateJobStatus call (#4198) Fast follow to #4191 Signed-off-by: Spencer Peterson <spencerjp@google.com> * Add support for Ray token auth (#4179) * Add support for Ray token auth Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * add e2e test for Ray cluster auth Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * address nits from Ruiean Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * update RAY_auth_mode -> RAY_AUTH_MODE Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * configure auth for Ray autoscaler Signed-off-by: Andrew Sy Kim <andrewsy@google.com> --------- Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * Bump js-yaml from 4.1.0 to 4.1.1 in /dashboard (#4194) Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.0 to 4.1.1. - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@4.1.0...4.1.1) --- updated-dependencies: - dependency-name: js-yaml dependency-version: 4.1.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * update minimum Ray version required for token authentication to 2.52.0 (#4201) * update minimum Ray version required for token authentication to 2.52.0 Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * update RayCluster auth e2e test to use Ray v2.52 Signed-off-by: Andrew Sy Kim <andrewsy@google.com> --------- Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * add samples for RayCluster token auth (#4200) Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * update (#4208) Signed-off-by: Future-Outlier <eric901201@gmail.com> * [RayJob] Add token authentication support for All mode (#4210) * dashboard client authentication support Signed-off-by: Future-Outlier <eric901201@gmail.com> * support rayjob Signed-off-by: Future-Outlier <eric901201@gmail.com> * update to fix api serverr err Signed-off-by: Future-Outlier <eric901201@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * updarte Signed-off-by: Future-Outlier <eric901201@gmail.com> * Rayjob sidecar mode auth token mode support Signed-off-by: Future-Outlier <eric901201@gmail.com> * RayJob support k8s job mode Signed-off-by: Future-Outlier <eric901201@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * Address Andrew's advice Signed-off-by: Future-Outlier <eric901201@gmail.com> * add todo x-ray-authorization comments Signed-off-by: Future-Outlier <eric901201@gmail.com> --------- Signed-off-by: Future-Outlier <eric901201@gmail.com> * [RayCluster] Enable Secret informer watch/list and remove unused RBAC verbs (#4202) * Add authentication secret reconciliation support Signed-off-by: Future-Outlier <eric901201@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * update Signed-off-by: Future-Outlier <eric901201@gmail.com> * fix flaky test Signed-off-by: Future-Outlier <eric901201@gmail.com> * remove test fix Signed-off-by: Rueian <rueiancsie@gmail.com> --------- Signed-off-by: Future-Outlier <eric901201@gmail.com> Signed-off-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Rueian <rueiancsie@gmail.com> * [APIServer][Docs] Add user guide for retry behavior & configuration (#4144) * [Docs] Add the draft description about feature intro, configurations, and usecases Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Fix] Update the retry walk-through Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Doc] rewrite the first 2 sections Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Doc] Revise documentation wording and add Observing Retry Behavior section Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Fix] fix linting issue by running pre-commit run berfore commiting Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Fix] fix linting errors in the Markdown linting Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Fix] Clean up the math equation Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * Update the math formula of Backoff calculation. Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Signed-off-by: JustinYeh <justinyeh1995@gmail.com> * [Fix] Explicitly mentioned exponential backoff and removed the customization parts Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * [Docs] Clarify naming by replacing “APIServer” with “KubeRay APIServer” Co-authored-by: Cheng-Yeh Chung <kenchung285@gmail.com> Signed-off-by: JustinYeh <justinyeh1995@gmail.com> * [Docs] Rename retry-configuration.md to retry-behavior.md for accuracy Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> * Update Title to KubeRay APIServer Retry Behavior Co-authored-by: Cheng-Yeh Chung <kenchung285@gmail.com> Signed-off-by: JustinYeh <justinyeh1995@gmail.com> * [Docs] Add a note about the limitation of retry configuration Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> --------- Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> Signed-off-by: JustinYeh <justinyeh1995@gmail.com> Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Co-authored-by: Cheng-Yeh Chung <kenchung285@gmail.com> * Support X-Ray-Authorization fallback header for accepting auth token via proxy (#4213) * Support X-Ray-Authorization fallback header for accepting auth token in dashboard Signed-off-by: Future-Outlier <eric901201@gmail.com> * remove todo comment Signed-off-by: Future-Outlier <eric901201@gmail.com> --------- Signed-off-by: Future-Outlier <eric901201@gmail.com> * [RayCluster] make auth token secret name consistency (#4216) Signed-off-by: fscnick <fscnick.dev@gmail.com> * [RayCluster] Status includes head containter status message (#4196) * [RayCluster] Status includes head containter status message Signed-off-by: Spencer Peterson <spencerjp@google.com> * lint Signed-off-by: Spencer Peterson <spencerjp@google.com> * [RayCluster] Containers not ready status reflects structured reason Signed-off-by: Spencer Peterson <spencerjp@google.com> * nit Signed-off-by: Spencer Peterson <spencerjp@google.com> --------- Signed-off-by: Spencer Peterson <spencerjp@google.com> * Remove erroneous call in applyServeTargetCapacity (#4212) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> * [RayJob] Add token authentication support for light weight job submitter (#4215) * [RayJob] light weight job submitter auth token support Signed-off-by: Future-Outlier <eric901201@gmail.com> * X-Ray-Authorization Signed-off-by: Rueian <rueiancsie@gmail.com> --------- Signed-off-by: Future-Outlier <eric901201@gmail.com> Signed-off-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Rueian <rueiancsie@gmail.com> * feat: kubectl ray get token command (#4218) * feat: kubectl ray get token command Signed-off-by: Rueian <rueiancsie@gmail.com> * Update kubectl-plugin/pkg/cmd/get/get_token_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Rueian <rueiancsie@gmail.com> * Update kubectl-plugin/pkg/cmd/get/get_token.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Rueian <rueiancsie@gmail.com> * make sure the raycluster exists before getting the secret Signed-off-by: Rueian <rueiancsie@gmail.com> * better ux Signed-off-by: Rueian <rueiancsie@gmail.com> * Update kubectl-plugin/pkg/cmd/get/get_token.go Co-authored-by: Han-Ju Chen (Future-Outlier) <eric901201@gmail.com> Signed-off-by: Rueian <rueiancsie@gmail.com> --------- Signed-off-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Han-Ju Chen (Future-Outlier) <eric901201@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com> Signed-off-by: Ping <fourhundredping@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Andrew Sy Kim <andrewsy@google.com> Signed-off-by: Kai-Hsun Chen <khchen@x.ai> Signed-off-by: Neo Chien <6762509+cchung100m@users.noreply.github.com> Signed-off-by: justinyeh1995 <justinyeh1995@gmail.com> Signed-off-by: win5923 <ken89@kimo.com> Signed-off-by: alimaazamat <alima.azamat2003@gmail.com> Signed-off-by: Spencer Peterson <spencerjp@google.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Rueian <rueiancsie@gmail.com> Signed-off-by: JustinYeh <justinyeh1995@gmail.com> Signed-off-by: fscnick <fscnick.dev@gmail.com> Co-authored-by: Ping <fourhundredping@gmail.com> Co-authored-by: Han-Ju Chen (Future-Outlier) <eric901201@gmail.com> Co-authored-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Kai-Hsun Chen <kaihsun@anyscale.com> Co-authored-by: Kavish <141061817+kash2104@users.noreply.github.com> Co-authored-by: Neo Chien <6762509+cchung100m@users.noreply.github.com> Co-authored-by: cchung100m <cchung100m@users.noreply.github.com> Co-authored-by: JustinYeh <justinyeh1995@gmail.com> Co-authored-by: Jun-Hao Wan <ken89@kimo.com> Co-authored-by: Divyam Raj <41264059+divyamraj18@users.noreply.github.com> Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Co-authored-by: Alima Azamat <92766804+alimaazamat@users.noreply.github.com> Co-authored-by: Spencer Peterson <spencerjp@google.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Cheng-Yeh Chung <kenchung285@gmail.com> Co-authored-by: fscnick <6858627+fscnick@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
When deploying a RayJob, the RayCluster status may have valuable info about runtime failures before the cluster becomes ready. In such cases, the RayJob remains Initializing and never updates its status.
This is a common failure mode and may confuse new users that don't realize they need to inspect the ray cluster -- but we should use the existing ray cluster status subfield to reflect this information.
Related issue number
n/a
Checks