Join Ray Jobs API `JobInfo` with GCS `JobTableData` #31046

architkulkarni · 2022-12-12T23:16:19Z

Signed-off-by: Archit Kulkarni architkulkarni@users.noreply.github.com

Why are these changes needed?

Add a new protobuf for JobInfo from the Ray Job API
Augment the existing GCS GetAllJobInfo endpoint to return this information, if available (not all GCS jobs were submitted via the Ray Job API; these jobs won't have this extra JobInfo.)

Related issue number

Closes #29621

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

stale · 2023-01-15T16:46:25Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

…-job-info-gcs

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

…-job-info-gcs

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

…-job-info-gcs Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

architkulkarni · 2023-01-25T21:37:20Z

@iycheng Could you please help give a preliminary review of the change in the GCS Job manager?

Currently all it's doing is to pull Ray Job API JobInfo from the internal KV for each job submitted via the Ray Job API. Because the internal KV get is async, we need some custom logic here involving a counter, but perhaps you know a better pattern to use. I worked on this with @rickyyx and @edoakes.

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

…-job-info-gcs

fishbone

LG!

fishbone · 2023-01-31T22:17:31Z

src/ray/gcs/gcs_server/gcs_job_manager.cc

@@ -146,13 +147,76 @@ void GcsJobManager::HandleGetAllJobInfo(rpc::GetAllJobInfoRequest request,
                                        rpc::GetAllJobInfoReply *reply,
                                        rpc::SendReplyCallback send_reply_callback) {
  RAY_LOG(INFO) << "Getting all job info.";
-  auto on_done = [reply, send_reply_callback](
+
+  int limit = std::numeric_limits<int>::max();


A more fundamental problem is that we need pagination for GetAll apis, otherwise, the backend still doing a lot of work.

architkulkarni · 2023-02-01T00:15:41Z

Test failures:

rllib multi_agent_independent_learning broken on master, unrelated
Datasets air:test_tensor_extension was broken on master, unrelated
Workflows test_basic_workflows_2 flaky on master, unrelated
serve:tutorial_rllib was broken on master

Java test failed, but it isn't failing on the flaky test tracker. Retrying it to be safe

architkulkarni · 2023-02-01T02:08:05Z

Java test passed, merging

…ay-project#31046)" This reverts commit b2c5e63.

…GetAllJobInfo endpoint (#32388) The changes to the GetAllJobInfo endpoint in #31046 did not handle the possibility that multiple job table jobs (drivers) could have the same submission_id. This can actually happen, for example if there are multiple ray.init() calls in a Ray Job API entrypoint command. The GCS would crash in this case due to failing a RAY_CHECK that the number of jobs equaled the number of submission_ids seen. This PR updates the endpoint to handle the above possibility, and adds a unit test which fails without this PR. Related issue number Closes #32213

…GetAllJobInfo endpoint (ray-project#32388) The changes to the GetAllJobInfo endpoint in ray-project#31046 did not handle the possibility that multiple job table jobs (drivers) could have the same submission_id. This can actually happen, for example if there are multiple ray.init() calls in a Ray Job API entrypoint command. The GCS would crash in this case due to failing a RAY_CHECK that the number of jobs equaled the number of submission_ids seen. This PR updates the endpoint to handle the above possibility, and adds a unit test which fails without this PR. Related issue number Closes ray-project#32213

…GetAllJobInfo endpoint (#32388) (#32426) The changes to the GetAllJobInfo endpoint in #31046 did not handle the possibility that multiple job table jobs (drivers) could have the same submission_id. This can actually happen, for example if there are multiple ray.init() calls in a Ray Job API entrypoint command. The GCS would crash in this case due to failing a RAY_CHECK that the number of jobs equaled the number of submission_ids seen. This PR updates the endpoint to handle the above possibility, and adds a unit test which fails without this PR. Related issue number Closes #32213 Co-authored-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

…ct#31046) Why are these changes needed? Add a new protobuf for JobInfo from the Ray Job API Augment the existing GCS GetAllJobInfo endpoint to return this information, if available (not all GCS jobs were submitted via the Ray Job API; these jobs won't have this extra JobInfo.) Related issue number Closes ray-project#29621 Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…GetAllJobInfo endpoint (ray-project#32388) The changes to the GetAllJobInfo endpoint in ray-project#31046 did not handle the possibility that multiple job table jobs (drivers) could have the same submission_id. This can actually happen, for example if there are multiple ray.init() calls in a Ray Job API entrypoint command. The GCS would crash in this case due to failing a RAY_CHECK that the number of jobs equaled the number of submission_ids seen. This PR updates the endpoint to handle the above possibility, and adds a unit test which fails without this PR. Related issue number Closes ray-project#32213 Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

This PR propagates the existing num_pending_tasks property of a driver's core worker process to a new boolean is_running_tasks field for that driver in the GCS Job table. Exposing this field is useful for external cluster managers to determine whether a cluster is "active" or not. The code for the new NumPendingTasks RPC from the GCS job manager to the driver's core worker process mimics the existing PushTask RPC made from the GCS actor manager to a worker's core worker process. The core worker client factory pattern is reused as well. To make the connection to the core worker from the Job manager, we update the Job table proto to include the full Address object (including the port) instead of just the driver IP address. This PR also adds unit tests in C++ and an end to end Python test. Related issue number Followup to PR #31046. Closes #30436

This PR propagates the existing num_pending_tasks property of a driver's core worker process to a new boolean is_running_tasks field for that driver in the GCS Job table. Exposing this field is useful for external cluster managers to determine whether a cluster is "active" or not. The code for the new NumPendingTasks RPC from the GCS job manager to the driver's core worker process mimics the existing PushTask RPC made from the GCS actor manager to a worker's core worker process. The core worker client factory pattern is reused as well. To make the connection to the core worker from the Job manager, we update the Job table proto to include the full Address object (including the port) instead of just the driver IP address. This PR also adds unit tests in C++ and an end to end Python test. Related issue number Followup to PR ray-project#31046. Closes ray-project#30436 Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

Add submission_id to JobInfo

8fcbc9f

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

stale bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jan 15, 2023

Merge branch 'master' of https://github.com/ray-project/ray into join…

18ca6e5

…-job-info-gcs

stale bot removed the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jan 23, 2023

architkulkarni added 10 commits January 23, 2023 07:11

Fix typo in entrypoint comment

426d34e

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Merge branch 'master' of https://github.com/ray-project/ray into join…

79e50f1

…-job-info-gcs

Serialize runtime_env to JSON

12f0f3a

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Fix issue with duplicate submission_id

f3daeac

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Add JobsAPIInfo

243d96d

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Query JobInfo from internal kv in GCS Job Manager

68fdb1f

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Query JobInfo from internal kc in GCS Job Manager

a932f44

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Merge branch 'master' of https://github.com/ray-project/ray into join…

e910e67

…-job-info-gcs Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Lint

c860838

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Undo doc change

28e14b0

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

architkulkarni assigned edoakes, rickyyx and fishbone Jan 25, 2023

architkulkarni added 6 commits January 25, 2023 13:38

Fix error with capture of job_submission_id

44ab6a4

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Load info into reply

c0426b7

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Lint

9c6ae8a

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Fix existing JobManager test

63b98a7

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Fix duplicate submission_id in JobDetails

ae6db4c

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

Revert add submission_id to JobInfo

04da99f

Signed-off-by: Archit Kulkarni <architkulkarni@users.noreply.github.com>

architkulkarni changed the title ~~[WIP] Join Ray Jobs API JobInfo with GCS JobTableData~~ Join Ray Jobs API JobInfo with GCS JobTableData Jan 26, 2023

architkulkarni marked this pull request as ready for review January 26, 2023 15:44

architkulkarni requested review from wuisawesome, ericl and AmeerHajAli as code owners January 26, 2023 15:44

architkulkarni added tests-ok The tagger certifies test failures are unrelated and assumes personal liability. and removed @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. labels Jan 31, 2023

Merge branch 'master' of https://github.com/ray-project/ray into join…

005983f

…-job-info-gcs

fishbone approved these changes Jan 31, 2023

View reviewed changes

fishbone reviewed Jan 31, 2023

View reviewed changes

architkulkarni merged commit b2c5e63 into ray-project:master Feb 1, 2023

architkulkarni mentioned this pull request Feb 6, 2023

[Ray 2.3 Release] Common failure in a few release tests failed to connect to all addresses; last error: UNKNOWN #32213

Closed

scv119 mentioned this pull request Feb 7, 2023

[Ray 2.3 Release] stress_test_many_tasks.json stage_3_time has 17% regression in 2.3.0 #32235

Closed

architkulkarni added a commit to architkulkarni/ray that referenced this pull request Feb 7, 2023

Revert "[Core] Join Ray Jobs API JobInfo with GCS JobTableData (r…

795d081

…ay-project#31046)" This reverts commit b2c5e63.

architkulkarni mentioned this pull request Feb 10, 2023

[Job API] Handle multiple drivers with same job submission id in GCS GetAllJobInfo endpoint #32388

Merged

7 tasks

architkulkarni mentioned this pull request May 10, 2023

[Core] Add is_running_tasks bit in JobStatus #35188

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Join Ray Jobs API `JobInfo` with GCS `JobTableData` #31046

Join Ray Jobs API `JobInfo` with GCS `JobTableData` #31046

architkulkarni commented Dec 12, 2022 •

edited

Loading

stale bot commented Jan 15, 2023

architkulkarni commented Jan 25, 2023 •

edited

Loading

fishbone left a comment

fishbone Jan 31, 2023

architkulkarni commented Feb 1, 2023

architkulkarni commented Feb 1, 2023

Join Ray Jobs API JobInfo with GCS JobTableData #31046

Join Ray Jobs API JobInfo with GCS JobTableData #31046

Conversation

architkulkarni commented Dec 12, 2022 • edited Loading

Why are these changes needed?

Related issue number

Checks

stale bot commented Jan 15, 2023

architkulkarni commented Jan 25, 2023 • edited Loading

fishbone left a comment

Choose a reason for hiding this comment

fishbone Jan 31, 2023

Choose a reason for hiding this comment

architkulkarni commented Feb 1, 2023

architkulkarni commented Feb 1, 2023

Join Ray Jobs API `JobInfo` with GCS `JobTableData` #31046

Join Ray Jobs API `JobInfo` with GCS `JobTableData` #31046

architkulkarni commented Dec 12, 2022 •

edited

Loading

architkulkarni commented Jan 25, 2023 •

edited

Loading