Adding VMware platforms support such as vSphere to Ray Autoscaler #37815

Shubhamurkade · 2023-07-26T16:15:59Z

Why are these changes needed?

Add support for ray to run on vSphere

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Shubham Urkade <surkade@vmware.com>

architkulkarni

Hi @Shubhamurkade, thanks for the work you've put into the PR. The PR is quite large and that makes it hard to review thoroughly. It would be really helpful if you could split the PR into a series of smaller PRs. Do you see a good way to split it up?

architkulkarni · 2023-07-26T18:04:47Z

doc/source/cluster/vms/getting-started.rst

+    .. tab-item:: vSphere
+
+        .. code-block:: shell
+
+            $ pip install vsphere-automation-sdk-python
+


Suggested change

.. tab-item:: vSphere

.. code-block:: shell

$ pip install vsphere-automation-sdk-python

.. tab-item:: vSphere (Experimental)

.. code-block:: shell

$ pip install -U "ray[default]" vsphere-automation-sdk-python

Would opt for labeling this as experimental for now until the feature sees more users and becomes more stable.

architkulkarni · 2023-07-26T18:06:06Z

doc/source/cluster/vms/getting-started.rst

+            $ export VSPHERE_SERVER = 192.168.0.1 # Enter your vSphere IP
+            $ export VSPHERE_USER = user # Enter your user name
+            $ export VSPHERE_PASSWORD = password # Enter your password
+


Might be good to link to relevant vSphere documentation here, similar to how we link to GCP docs for credentials above

Hi @architkulkarni There's no standard way to provide credentials to vSphere. Our code just expects the user to provide these credentials as env variables.

Shubhamurkade · 2023-07-27T07:24:23Z

Hi @Shubhamurkade, thanks for the work you've put into the PR. The PR is quite large and that makes it hard to review thoroughly. It would be really helpful if you could split the PR into a series of smaller PRs. Do you see a good way to split it up?

Hi @architkulkarni The majority of the changes are in python/ray/autoscaler/_private/vsphere directory. These are the minimum subset required to achieve the ray on vSphere support. I could do the following:

Move documentation update to another PR.
Move python/ray/autoscaler/vsphere/defaults.yaml to another PR.
Write a detailed documentation which can explain the code flow in python/ray/autoscaler/_private/vsphere directory.

Please let me know if you would need more support.

architkulkarni · 2023-07-27T21:03:27Z

I see. In that case, I like your idea 3--would you mind adding some documentation, perhaps in an ARCHITECTURE.md file in the python/ray/autoscaler/_private/vsphere directory? You could emphasize the vSphere-specific parts which might not be as familiar to Ray developers. I didn't find any similar documentation for the other cloud cluster launchers, but here are some examples from other parts of the code in case they're helpful: https://github.com/ray-project/ray/blob/master/python/ray/util/client/ARCHITECTURE.md, https://github.com/ray-project/ray/blob/master/python/ray/runtime_env/ARCHITECTURE.md

No need to split the docs and sample YAML into another PR. I'll review this PR piece by piece but may not have time to review all of it at once.

architkulkarni

I've reviewed up to create_node in node_provider.py, only minor comments so far.

doc/source/cluster/vms/getting-started.rst

python/ray/autoscaler/_private/command_runner.py

doc/source/cluster/vms/getting-started.rst

architkulkarni · 2023-07-27T21:37:41Z

python/ray/autoscaler/_private/vsphere/node_provider.py

+            vsphere_credentials["user"],
+            vsphere_credentials["password"],
+            VmwSdkClient.SessionType.UNVERIFIED,
+            VmwSdkClient.ClientType.PYVMOMI_SDK,


What's the difference between automation and pyvmomi?

automation sdk uses REST interface & Pyvmomi is supported by VMware SOAP APIs.
We are currently using APIs from both. We plan to move to either of them once we identify the necessary support.

python/ray/autoscaler/_private/vsphere/node_provider.py

Signed-off-by: Chen Hui <huchen@vmware.com>

update the test-script to support test ray-on-vsphere

instant clone only, and address review comments

architkulkarni · 2023-08-07T23:40:13Z

Thanks for adding the doc, will continue the review shortly!

fix issues

JingChen23 · 2023-08-14T09:58:37Z

Oh I think you need to add the test to https://github.com/ray-project/ray/blob/master/python/ray/tests/BUILD#L258 for it to run in our CI, would you mind adding it?

Added, thanks for pointing that out.

krfricke · 2023-08-14T13:22:10Z

Attention: External code changed

This PR changes code that is used or cited in external sources, e.g. blog posts.

Before merging this PR, please make sure that the code in the external sources is still working, and consider updating them to reflect the changes.

The affected files and the external sources are:
* `doc/external/pytorch_tutorials_hyperparameter_tuning_tutorial.py`: https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html

It looks like this got triggered from the force-push. It doesn't look like the file was actually changed, so from an external code perspective, we can go ahead.

architkulkarni · 2023-08-14T17:07:10Z

Any ideas about this build failure? Not sure what could be causing it off the top of my head. https://buildkite.com/ray-project/oss-ci-build-pr/builds/31726#0189f389-1c82-452a-a638-1c0b7a963aec/5460-5485

Happens to several (all?) python tests: https://buildkite.com/ray-project/oss-ci-build-pr/builds/31726#0189f389-1c6b-4ef2-98e4-a1a14c9876e8/1741-1781

architkulkarni · 2023-08-14T17:08:32Z

@scv119 @rickyyx do either of you plan to review this PR? It needs codeowner approval from ray-core.

architkulkarni · 2023-08-14T17:11:18Z

python/ray/tests/BUILD

+py_test(
+    name = "test_vsphere_node_provider",
+    size = "small",
+    srcs = ["gcp/test_vsphere_node_provider.py"],


Suggested change

srcs = ["gcp/test_vsphere_node_provider.py"],

srcs = ["vsphere/test_vsphere_node_provider.py"],

I think this is causing the build failure

Oh gosh, will fix this right now.

Fxied in the latest commit, thanks for helping us do the troubleshooting!

Signed-off-by: Chen Jing <jingch@vmware.com>

JingChen23 · 2023-08-14T23:51:54Z

Attention: External code changed

This PR changes code that is used or cited in external sources, e.g. blog posts.
Before merging this PR, please make sure that the code in the external sources is still working, and consider updating them to reflect the changes.
The affected files and the external sources are:
* `doc/external/pytorch_tutorials_hyperparameter_tuning_tutorial.py`: https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html
It looks like this got triggered from the force-push. It doesn't look like the file was actually changed, so from an external code perspective, we can go ahead.

Yes, we used to have one PR tried to fixed the DCO issue, but we failed, so did a force push to revert, to avoid messing up the commits.

Signed-off-by: Chen Jing <jingch@vmware.com>

architkulkarni · 2023-08-16T16:58:47Z

Failed tests:

windows://python/ray/tests:test_autoscaler_e2e flaky on https://flaky-tests.ray.io/ and unrelated
//rllib:TestLearnerGroupAsyncUpdate unrelated
//rllib:test_memory_leak_ppo unrelated
//python/ray/tune:test_trial_scheduler unrelated
//doc:external/pytorch_tutorials_hyperparameter_tuning_tutorial unrelated
//python/ray/tests:test_client_builder unrelated
//python/ray/tests:test_out_of_disk_space unrelated

…y-project#37815) --------- Signed-off-by: Shubham Urkade <surkade@vmware.com> Signed-off-by: Chen Hui <huchen@vmware.com> Signed-off-by: Chen Jing <jingch@vmware.com> Co-authored-by: Chen Hui <huchen@vmware.com> Co-authored-by: Chen Jing <jingch@vmware.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

Similar to other providers, we add example-minimal.yaml and example-full.yaml to vSphere autoscaler. And we add and refine vSphere related references in the Getting Started guide as well as the cluster configuration reference page, based on the newly added examples. Why are these changes needed? In PR #37815 we've added vSphere platform support to Ray Autoscaler. However, the related documents are not sufficient. This follow-up change adds related examples similar to other platforms. The related documents including the getting-started guide as well as the cluster configuration reference also need to be updated to include descriptions specific for vSphere. We will do another follow-up PR to add a "Launching Ray Clusters on vSphere" user guide at https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/index.html Signed-off-by: Fangchi Wang <wfangchi@vmware.com>

…project#39379) Similar to other providers, we add example-minimal.yaml and example-full.yaml to vSphere autoscaler. And we add and refine vSphere related references in the Getting Started guide as well as the cluster configuration reference page, based on the newly added examples. Why are these changes needed? In PR ray-project#37815 we've added vSphere platform support to Ray Autoscaler. However, the related documents are not sufficient. This follow-up change adds related examples similar to other platforms. The related documents including the getting-started guide as well as the cluster configuration reference also need to be updated to include descriptions specific for vSphere. We will do another follow-up PR to add a "Launching Ray Clusters on vSphere" user guide at https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/index.html Signed-off-by: Fangchi Wang <wfangchi@vmware.com>

…) (#39399) Similar to other providers, we add example-minimal.yaml and example-full.yaml to vSphere autoscaler. And we add and refine vSphere related references in the Getting Started guide as well as the cluster configuration reference page, based on the newly added examples. Why are these changes needed? In PR #37815 we've added vSphere platform support to Ray Autoscaler. However, the related documents are not sufficient. This follow-up change adds related examples similar to other platforms. The related documents including the getting-started guide as well as the cluster configuration reference also need to be updated to include descriptions specific for vSphere. We will do another follow-up PR to add a "Launching Ray Clusters on vSphere" user guide at https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/index.html Signed-off-by: Fangchi Wang <wfangchi@vmware.com> Co-authored-by: Fangchi Wang <wfangchi@vmware.com>

…project#39379) Similar to other providers, we add example-minimal.yaml and example-full.yaml to vSphere autoscaler. And we add and refine vSphere related references in the Getting Started guide as well as the cluster configuration reference page, based on the newly added examples. Why are these changes needed? In PR ray-project#37815 we've added vSphere platform support to Ray Autoscaler. However, the related documents are not sufficient. This follow-up change adds related examples similar to other platforms. The related documents including the getting-started guide as well as the cluster configuration reference also need to be updated to include descriptions specific for vSphere. We will do another follow-up PR to add a "Launching Ray Clusters on vSphere" user guide at https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/index.html Signed-off-by: Fangchi Wang <wfangchi@vmware.com> Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>

…y-project#37815) --------- Signed-off-by: Shubham Urkade <surkade@vmware.com> Signed-off-by: Chen Hui <huchen@vmware.com> Signed-off-by: Chen Jing <jingch@vmware.com> Co-authored-by: Chen Hui <huchen@vmware.com> Co-authored-by: Chen Jing <jingch@vmware.com> Signed-off-by: Victor <vctr.y.m@example.com>

…project#39379) Similar to other providers, we add example-minimal.yaml and example-full.yaml to vSphere autoscaler. And we add and refine vSphere related references in the Getting Started guide as well as the cluster configuration reference page, based on the newly added examples. Why are these changes needed? In PR ray-project#37815 we've added vSphere platform support to Ray Autoscaler. However, the related documents are not sufficient. This follow-up change adds related examples similar to other platforms. The related documents including the getting-started guide as well as the cluster configuration reference also need to be updated to include descriptions specific for vSphere. We will do another follow-up PR to add a "Launching Ray Clusters on vSphere" user guide at https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/index.html Signed-off-by: Fangchi Wang <wfangchi@vmware.com> Signed-off-by: Victor <vctr.y.m@example.com>

Support Ray on vSphere

322a15e

Signed-off-by: Shubham Urkade <surkade@vmware.com>

Shubhamurkade requested review from architkulkarni, wuisawesome, DmitriGekhtman, maxpumperla, pcmoritz, a team and ericl as code owners July 26, 2023 16:16

Shubhamurkade force-pushed the ray-vmware-support branch from dc571fc to 322a15e Compare July 26, 2023 16:26

architkulkarni assigned gvspraveen, scv119 and architkulkarni Jul 26, 2023

akshay-anyscale requested a review from scv119 July 26, 2023 20:18

architkulkarni reviewed Jul 26, 2023

View reviewed changes

architkulkarni assigned richardliaw Jul 27, 2023

architkulkarni reviewed Jul 27, 2023

View reviewed changes

Chen Hui and others added 8 commits August 1, 2023 08:05

update the test-script to support test ray-on-vsphere

2f2ebf6

Signed-off-by: Chen Hui <huchen@vmware.com>

Follow a consistent programming style

39b0fa2

Signed-off-by: Chen Hui <huchen@vmware.com>

make the code consistent with our pr to Ray and redo lint

aa48046

Merge pull request #1 from huchen2021/support-test-script-vsphere

aebaac1

update the test-script to support test ray-on-vsphere

Merge pull request #2 from JingChen23/address-pr-comments

bee3086

instant clone only, and address review comments

Remove logs from non_terminated_nodes

8ad836b

Merge upstream branch

06294a0

Add ARCHITECTURE.md file

6149258

JingChen23 and others added 2 commits August 8, 2023 08:23

fix issues

40214ea

Merge pull request #3 from JingChen23/fix-issues

20b9333

fix issues

architkulkarni reviewed Aug 14, 2023

View reviewed changes

fix path error on the Build file

003fed8

Signed-off-by: Chen Jing <jingch@vmware.com>

JingChen23 added 6 commits August 15, 2023 10:21

add vsphere automation sdk in to the requirements for test

4f13ff6

Signed-off-by: Chen Jing <jingch@vmware.com>

add vsphere automation sdk in to the requirements for test

a8b2bc2

Signed-off-by: Chen Jing <jingch@vmware.com>

Bypass the windows tests and fix one little issue

68dff61

Signed-off-by: Chen Jing <jingch@vmware.com>

Merge branch 'master' into ray-vmware-support

4fc94a6

Signed-off-by: Chen Jing <jingch@vmware.com>

fix the test name in BUILD to be the same in ci.sh

89700de

Signed-off-by: Chen Jing <jingch@vmware.com>

fix the test name in ci.sh

c80dbd6

Signed-off-by: Chen Jing <jingch@vmware.com>

Shubhamurkade changed the title ~~Support Ray on vSphere~~ Adding VMware platforms support such as vSphere to Ray Autoscaler Aug 16, 2023

rickyyx approved these changes Aug 16, 2023

View reviewed changes

architkulkarni added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Aug 16, 2023

architkulkarni merged commit 4428a3f into ray-project:master Aug 16, 2023

wfangchi mentioned this pull request Sep 7, 2023

[Doc] Add vSphere cluster configuration reference with examples #39379

Merged

8 tasks

architkulkarni mentioned this pull request Sep 7, 2023

[Doc] Add vSphere cluster configuration reference with examples (#39379) #39399

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding VMware platforms support such as vSphere to Ray Autoscaler #37815

Adding VMware platforms support such as vSphere to Ray Autoscaler #37815

Shubhamurkade commented Jul 26, 2023

architkulkarni left a comment

architkulkarni Jul 26, 2023

architkulkarni Jul 26, 2023

architkulkarni Jul 26, 2023

Shubhamurkade Aug 4, 2023

Shubhamurkade commented Jul 27, 2023

architkulkarni commented Jul 27, 2023 •

edited

Loading

architkulkarni left a comment

architkulkarni Jul 27, 2023

vinodkri Aug 1, 2023

architkulkarni commented Aug 7, 2023

JingChen23 commented Aug 14, 2023

krfricke commented Aug 14, 2023

Attention: External code changed

architkulkarni commented Aug 14, 2023 •

edited

Loading

architkulkarni commented Aug 14, 2023

architkulkarni Aug 14, 2023

architkulkarni Aug 14, 2023

JingChen23 Aug 14, 2023

JingChen23 Aug 14, 2023

JingChen23 commented Aug 14, 2023

Attention: External code changed

architkulkarni commented Aug 16, 2023

	srcs = ["gcp/test_vsphere_node_provider.py"],
	srcs = ["vsphere/test_vsphere_node_provider.py"],

Adding VMware platforms support such as vSphere to Ray Autoscaler #37815

Adding VMware platforms support such as vSphere to Ray Autoscaler #37815

Conversation

Shubhamurkade commented Jul 26, 2023

Why are these changes needed?

Related issue number

Checks

architkulkarni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shubhamurkade commented Jul 27, 2023

architkulkarni commented Jul 27, 2023 • edited Loading

architkulkarni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

architkulkarni commented Aug 7, 2023

JingChen23 commented Aug 14, 2023

krfricke commented Aug 14, 2023

Attention: External code changed

architkulkarni commented Aug 14, 2023 • edited Loading

architkulkarni commented Aug 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JingChen23 commented Aug 14, 2023

Attention: External code changed

architkulkarni commented Aug 16, 2023

architkulkarni commented Jul 27, 2023 •

edited

Loading

architkulkarni commented Aug 14, 2023 •

edited

Loading