Speed up refresh: delay the slower ray status call & use cached IPs.
#2079
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Michaelvll
approved these changes
Jun 13, 2023
There was a problem hiding this comment.
Thanks for the optimization and benchmark @concretevitamin! The changes look good to me.
Comment on lines
+330
to
+337
| # accelerator_args is way too long. | ||
| # Convert from: | ||
| # GCP(n1-highmem-8, {'tpu-v2-8': 1}, accelerator_args={'runtime_version': '2.5.0'} # pylint: disable=line-too-long | ||
| # to: | ||
| # GCP(n1-highmem-8, {'tpu-v2-8': 1}...) | ||
| pattern = ', accelerator_args={.*}' | ||
| launched_resource_str = re.sub(pattern, '...', | ||
| launched_resource_str) |
There was a problem hiding this comment.
Seems if we specify the disk_type, disk_tier, etc, they will be after .... Is that intended?
There was a problem hiding this comment.
Yes, it'd show GCP(n1-highmem-8, {'tpu-v2-8': 1}..., cpus=2+, dist_tier=high), etc. Wdyt?
Michaelvll
approved these changes
Jun 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I noticed that
status -ris taking way too long. With some investigation with @Michaelvll, this PR Implemented two optimizations:ray statuscallAll tests below are 1-node on-demand GCP clusters. These are all "normal" case where the runtime on the cluster did not become problematic.
handle.external_ips(use_cached_ips=False) -> ray get head-ip/worker-ipsis too slow; replace with NodeProvider calls or cloud CLI/SDK calls?Results:
do
ray statuslast + use_cached_ips=TrueSTOPPED -> STOPPED
UP, autostop not set -> UP
UP, autostopped -> STOPPED
UP, autostop set -> UP
INIT -> INIT
(if we do the first optimization only) do
ray statuslastSTOPPED -> STOPPED
UP, autostop not set -> UP
UP, autostopped -> STOPPED
UP, autostop set -> UP
INIT -> INIT
Tested (run the relevant ones):
bash format.shpytest tests/test_smoke.py:pytest tests/test_smoke.py --generic-cloud awspytest tests/test_smoke.py::test_fill_in_the_namebash tests/backward_comaptibility_tests.sh