[AWS] Failover Optimization from Quota #1953

shethhriday29 · 2023-05-12T00:59:31Z

Description:

Drastically reduce failover time when we attempt to provision accelerators in an AWS region where the region's quota for those accelerators is zero
This involved directly querying every region's accelerator quota immediately before provision, and if the query returns a 0 value, skipping the provision attempt (and potentially moving onto the next region as a result)
Used AWS's boto3 Python API for query
Reduces the failover in these zero-quota cases from 30 seconds (failed provisioning attempt) to <1 second (time for boto3 query to return)
Had to add a file to the catalog (skypilot-catalog, me/@shethhriday29 opened a corresponding PR there) to map accelerators to their quota codes, which are needed in the boto3 queries

Tested (run the relevant ones):

Ran all AWS smoke tests
Created new smoke test, test_aws_zero_quota_failover, that ensures that the zero quota checker works
Worked with Romil to test backward compatibility with new catalog changes
Manually tested functionality by attempting to (across multiple types of accelerators):

Provision resources in regions with zero quota, and ensuring that the automatic failover happens
Provision resources in non-zero but still not-adequate quota, and ensuring that the provision is still attempted
Provision resources in a region with adequate quota and ensuring the provision is successful

romilbhardwaj

Awesome work @shethhriday29, and welcome to SkyPilot! Took a quick glance and left some high-level comments. Will take a closer look later. Also recommend running ./format.sh to automatically format your code to keep the style checker happy.

sky/clouds/service_catalog/constants.py

sky/clouds/aws.py

sky/clouds/cloud.py

…uota-optimization

romilbhardwaj

Awesome work @shethhriday29! Left some comments.

sky/clouds/aws.py

sky/clouds/cloud.py

sky/backends/cloud_vm_ray_backend.py

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

sky/backends/cloud_vm_ray_backend.py

tests/test_smoke.py

romilbhardwaj

This looks ready to go! Thanks for the awesome work @shethhriday29 - this will make failover much faster for AWS! 🚀

Michaelvll

Thanks for adding this feature @shethhriday29! It is exciting to see the optimization. Left several nits about code style and robustness. The PR should be ready to go, after the comments are fixed and the tests are passed.

sky/backends/cloud_vm_ray_backend.py

Michaelvll · 2023-06-21T20:34:26Z

sky/clouds/aws.py

+                             region: str,
+                             instance_type: str,
+                             use_spot: bool = False) -> bool:
+        """


style nit: Having a single line description of this function in the first line.

Suggested change

"""

"""Check if the quota is available for the requested instance_type

Google styleguide: First line should be on the same line as the triple quotes. https://google.github.io/styleguide/pyguide.html#383-functions-and-methods

sky/clouds/aws.py

tests/test_smoke.py

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

Michaelvll

Thanks for the fix @shethhriday29! Left a final comment about the importing. Otherwise, it looks good to me.

sky/clouds/aws.py

tests/test_smoke.py

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

Michaelvll

Thanks for the update @shethhriday29! LGTM.

concretevitamin · 2023-06-23T16:19:36Z

This is awesome @shethhriday29. A few UX notes:

sky launch --gpus A100:8 --use-spot --down --cloud aws prints quota warnings like

W 06-23 09:09:45 cloud_vm_ray_backend.py:1947] sky.exceptions.ResourcesUnavailableError: Found no quota for p4d.24xlarge in region ap-northeast-2. To request quotas, check the instruction: https://skypilot.readthedocs.io/en/latest/cloud-setup/quota.html.

However, removing the --use-spot flag I could successfully provision.

A few ideas to improve this logging:

Say it's the spot quota that's out; not on-demand
Even better, would it be possible to print the precise quota text used in AWS quota console, such as All P Spot Instance Requests vs. Running On-Demand P instances?

In the quota warnings above, it may be better to make the warning yellow. Otherwise it's a bit hard to spot.

…uota-optimization

concretevitamin

Awesome @shethhriday29! LGTM.

concretevitamin · 2023-06-26T16:18:01Z

sky/clouds/aws.py

+                             region: str,
+                             instance_type: str,
+                             use_spot: bool = False) -> bool:
+        """


Google styleguide: First line should be on the same line as the triple quotes. https://google.github.io/styleguide/pyguide.html#383-functions-and-methods

sky/clouds/aws.py

concretevitamin · 2023-06-26T16:19:08Z

sky/clouds/cloud.py

+                              instance_type: str,
+                              use_spot: bool = False) -> bool:
+        """
+        Checks to ensure that a particular accelerator has a nonzero quota


ditto (styleguide)

tests/test_smoke.py

…uota-optimization

romilbhardwaj · 2023-06-26T22:47:23Z

Thanks @shethhriday29! Looks like this is ready to go in. Merging now!

Sheth and others added 10 commits April 18, 2023 01:42

initial commit

2d1bf15

more quick changes

d674726

try push

f91a3f1

remove test lines

164b490

quick file add

a647263

things finally working

8053c9f

Merge branch 'master' into aws-quota-optimization

c29da94

refactoring

63fd297

quick string fix

bc77487

removed test cases

b858a94

romilbhardwaj self-requested a review May 12, 2023 01:04

romilbhardwaj reviewed May 12, 2023

View reviewed changes

sky/clouds/service_catalog/constants.py Outdated Show resolved Hide resolved

sky/clouds/aws.py Show resolved Hide resolved

sky/clouds/cloud.py Show resolved Hide resolved

concretevitamin self-requested a review May 12, 2023 03:39

This was referenced May 13, 2023

[Feature request] How to check the GPU quota with Sky? #447

Closed

Can Sky prioritize regions with quota available? #664

Closed

Michaelvll mentioned this pull request May 28, 2023

[Provisioner][Feature Request] Use quotas to guide region/cloud fail over #1268

Closed

Sheth added 3 commits June 17, 2023 08:13

Merge branch 'master' of github.com:shethhriday29/skypilot into aws-q…

6bc102a

…uota-optimization

Merge branch 'master' of github.com:shethhriday29/skypilot into aws-q…

959dae4

…uota-optimization

formatting changes, comments, and docstring

fe2c0ff

romilbhardwaj approved these changes Jun 19, 2023

View reviewed changes

Sheth and others added 5 commits June 19, 2023 15:31

added test for aws zero quota failover

45dd0ff

Update sky/clouds/cloud.py

f472bf4

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

merge with master

8d04c00

edits based on romil review

4e843b8

ran formatter

2ffa694

romilbhardwaj reviewed Jun 20, 2023

View reviewed changes

sky/backends/cloud_vm_ray_backend.py Outdated Show resolved Hide resolved

sky/backends/cloud_vm_ray_backend.py Show resolved Hide resolved

tests/test_smoke.py Outdated Show resolved Hide resolved

romilbhardwaj reviewed Jun 20, 2023

View reviewed changes

tests/test_smoke.py Outdated Show resolved Hide resolved

Sheth added 2 commits June 19, 2023 19:55

more romil comments

669af4c

formatting

f6b8b69

romilbhardwaj approved these changes Jun 20, 2023

View reviewed changes

Michaelvll self-requested a review June 21, 2023 20:25

Michaelvll approved these changes Jun 21, 2023

View reviewed changes

Sheth and others added 4 commits June 21, 2023 21:16

zhanghao edits

1ee1d89

Update tests/test_smoke.py

91f22f8

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

Update sky/backends/cloud_vm_ray_backend.py

1b8bcc2

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

formatting

8aa8722

shethhriday29 changed the title ~~Aws quota optimization~~ [AWS] Failover Optimization from Quota Jun 22, 2023

Michaelvll reviewed Jun 22, 2023

View reviewed changes

sky/clouds/aws.py Outdated Show resolved Hide resolved

sky/clouds/aws.py Outdated Show resolved Hide resolved

sky/clouds/aws.py Outdated Show resolved Hide resolved

tests/test_smoke.py Outdated Show resolved Hide resolved

shethhriday29 and others added 5 commits June 22, 2023 23:29

Update tests/test_smoke.py

93d5f1d

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

Update sky/clouds/aws.py

4ae2214

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

Update sky/clouds/aws.py

8943ef1

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

took out imports

e2db07b

format

268255e

Michaelvll approved these changes Jun 23, 2023

View reviewed changes

Sheth added 3 commits June 24, 2023 09:17

better logging

b8b371c

Merge branch 'master' of github.com:shethhriday29/skypilot into aws-q…

f471738

…uota-optimization

formatting

ac54096

concretevitamin approved these changes Jun 26, 2023

View reviewed changes

shethhriday29 added 2 commits June 26, 2023 17:24

Merge branch 'master' of github.com:shethhriday29/skypilot into aws-q…

b5706eb

…uota-optimization

zongheng comments

0b0a76d

romilbhardwaj merged commit 26e30ce into skypilot-org:master Jun 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AWS] Failover Optimization from Quota #1953

[AWS] Failover Optimization from Quota #1953

shethhriday29 commented May 12, 2023 •

edited

Loading

romilbhardwaj left a comment

romilbhardwaj left a comment

romilbhardwaj left a comment

Michaelvll left a comment •

edited

Loading

Michaelvll Jun 21, 2023

concretevitamin Jun 26, 2023

Michaelvll left a comment

Michaelvll left a comment

concretevitamin commented Jun 23, 2023

concretevitamin left a comment

concretevitamin Jun 26, 2023

concretevitamin Jun 26, 2023

romilbhardwaj commented Jun 26, 2023

	"""
	"""Check if the quota is available for the requested instance_type

[AWS] Failover Optimization from Quota #1953

[AWS] Failover Optimization from Quota #1953

Conversation

shethhriday29 commented May 12, 2023 • edited Loading

romilbhardwaj left a comment

Choose a reason for hiding this comment

romilbhardwaj left a comment

Choose a reason for hiding this comment

romilbhardwaj left a comment

Choose a reason for hiding this comment

Michaelvll left a comment • edited Loading

Choose a reason for hiding this comment

Michaelvll Jun 21, 2023

Choose a reason for hiding this comment

concretevitamin Jun 26, 2023

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

concretevitamin commented Jun 23, 2023

concretevitamin left a comment

Choose a reason for hiding this comment

concretevitamin Jun 26, 2023

Choose a reason for hiding this comment

concretevitamin Jun 26, 2023

Choose a reason for hiding this comment

romilbhardwaj commented Jun 26, 2023

shethhriday29 commented May 12, 2023 •

edited

Loading

Michaelvll left a comment •

edited

Loading