[UX - SkyServe] user now is able to select a LB policy from a range of options #4061

AlexCuadron · 2024-10-10T09:32:04Z

I added the option to specify different LB policies, by default, RoundRobin is used. This PR is intended to enable easy switching between LB policies to facilitate its development.

The default behaviour (without user interaction) doesn't modify the execution flow and the user is not allowed to use any other LB policy other than round-robing without them being added explicitly.

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

cblmemo

This is awesome @AlexCuadron ! It will greatly expand our customizability. Left some discussion ;) We might also think of how to expose this feature to our end user, in our Service YAML - maybe add a load_balancing_policy section under service?

sky/serve/load_balancer.py

AlexCuadron · 2024-10-17T10:21:55Z

@cblmemo Done! PTAL again :D

cblmemo

Thanks @AlexCuadron for the awesome work! It mostly looks good to me. Left some discussions ;)

examples/serve/gorilla/gorilla.yaml

sky/serve/load_balancer.py

sky/serve/load_balancing_policies.py

sky/serve/service_spec.py

AlexCuadron · 2024-10-18T12:34:25Z

Thanks for the comments @cblmemo, fixed and ready for next round 💪

cblmemo

Thanks for the quick fix @AlexCuadron ! Mostly looks good to me. Left some discussions.

btw, I created a branch here and lets merge our PR to the branch first. Merging into master might need more time and we want to move fast ;)

https://github.com/skypilot-org/skypilot/tree/heterogeneous-lb

sky/serve/load_balancing_policies.py

AlexCuadron · 2024-10-20T06:13:22Z

I changed the base branch and updated based on comments :)
PTAL @cblmemo

cblmemo

Thanks for the prompt fix @AlexCuadron ! Mostly looks good to me. Left some nits :))

examples/serve/misc/cancel/service.yaml

examples/serve/llama2/llama2.yaml

sky/serve/load_balancer.py

sky/serve/service_spec.py

AlexCuadron · 2024-10-21T11:23:07Z

Done! PTAL @cblmemo :D

cblmemo

Thanks for the prompt fix @AlexCuadron ! Left some final nits 🚀

examples/serve/load_balancing_policies_example.yaml

sky/serve/load_balancer.py

sky/serve/__init__.py

Co-authored-by: Tian Xia <cblmemo@gmail.com>

* [Catalog] Silently ignore TPU price not found. * assert for non tpu v6e * format

…ing (skypilot-org#4264) * fix race condition for setting job status to FAILED during INIT * Fix * fix * format * Add smoke tests * revert pending submit * remove update entirely for the job schedule step * wait for job 32 to finish * fix smoke * move and rename * Add comment * minor

…-org#4278) Set worker minimum port number

* [docs] use k8s instead of kubernetes in the CLI * fix docs build script for linux * Update docs/source/reference/kubernetes/kubernetes-getting-started.rst Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com> --------- Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

* [jobs] autodown managed job clusters If all goes correctly, the managed job controller should tear down a managed job cluster once the managed job completes. However, if the controller fails somehow (e.g. crashes, is terminated, etc), we don't want to leak resources. As a failsafe, set autodown on the job cluster. This is not foolproof, since the skylet on the cluster can also crash, but it's likely to catch many cases. * add comment about autodown duration * add leading _

* Update cloud_vm_ray_backend.py * Update cloud_vm_ray_backend.py * format

…4274) fix: multiple `job_id`

AlexCuadron · 2024-11-08T00:48:37Z

PTAL @Michaelvll

cblmemo

Hi @AlexCuadron ! Thanks for adding this. I tried this PR and got the following:

$ sky serve up examples/serve/load_balancing_policies_example.yaml
Service from YAML spec: examples/serve/load_balancing_policies_example.yaml
ValueError: Invalid service YAML: Found unsupported field 'load_balancing_policy'.

Should we also update the sky/utils/schemas.py?

sky/serve/load_balancer.py

sky/serve/service_spec.py

Co-authored-by: Tian Xia <cblmemo@gmail.com>

1. Add available policies to schema validation 2. Show available policies in error message when invalid policy is specified 3. Display load balancing policy in service spec repr when explicitly set

…policies Only 'round_robin' is currently implemented in LoadBalancingPolicy class

sky/utils/schemas.py

Move policy validation to code to avoid duplication and make it easier to maintain when adding new policies

AlexCuadron · 2024-11-08T23:45:48Z

PTAL @cblmemo

cblmemo

Thanks for the quick fix @AlexCuadron ! Left 2 final nits ;)

sky/utils/schemas.py

sky/serve/service_spec.py

Co-authored-by: Tian Xia <cblmemo@gmail.com>

…mport inside function

AlexCuadron · 2024-11-10T02:49:01Z

Oops, sorry for the circular import 😅
Should be gtg @cblmemo 👍

cblmemo

Thanks for the prompt fix @AlexCuadron ! LGTM.

AlexCuadron added 3 commits October 10, 2024 11:20

user can select load balancing policies

bdf8a83

some fixes

c83444c

linting

1b01ced

cblmemo reviewed Oct 11, 2024

View reviewed changes

sky/serve/load_balancer.py Outdated Show resolved Hide resolved

sky/serve/load_balancer.py Outdated Show resolved Hide resolved

AlexCuadron added 4 commits October 17, 2024 11:34

Merge branch 'master' into user_select_policy

5995cd0

Fixes according to comments

1130d85

Linting

f1e771d

Linting

5755ab9

cblmemo reviewed Oct 17, 2024

View reviewed changes

AlexCuadron added 2 commits October 18, 2024 14:24

Merge branch 'master' into user_select_policy

1d170f6

Fixed according to comments

6cbd617

AlexCuadron requested a review from cblmemo October 19, 2024 19:07

cblmemo reviewed Oct 20, 2024

View reviewed changes

sky/serve/load_balancing_policies.py Show resolved Hide resolved

AlexCuadron changed the base branch from master to heterogeneous-lb October 20, 2024 06:09

fix

45d0f21

removed line from examples

f927265

cblmemo reviewed Oct 20, 2024

View reviewed changes

AlexCuadron added 5 commits October 21, 2024 12:53

Reverted changes

247f1c4

Merge branch 'master' into user_select_policy

6bfa60d

Reverted changes

872e34e

Fixed according to comments

4077656

Linting

1cf5ae3

AlexCuadron requested a review from cblmemo October 21, 2024 11:34

cblmemo reviewed Oct 23, 2024

View reviewed changes

examples/serve/load_balancing_policies_example.yaml Outdated Show resolved Hide resolved

sky/serve/load_balancer.py Outdated Show resolved Hide resolved

sky/serve/__init__.py Outdated Show resolved Hide resolved

AlexCuadron and others added 2 commits October 23, 2024 13:38

Update sky/serve/load_balancer.py

7517eaf

Co-authored-by: Tian Xia <cblmemo@gmail.com>

[Catalog] Silently ignore TPU price not found. (skypilot-org#4134)

0fba990

* [Catalog] Silently ignore TPU price not found. * assert for non tpu v6e * format

Michaelvll and others added 7 commits November 7, 2024 15:55

Set minimum port number a Ray worker can listen on to 11002 (skypilot…

60ad348

…-org#4278) Set worker minimum port number

[UX] Improve Formatting of Post Job Creation Logs (skypilot-org#4198)

c8ec7eb

* Update cloud_vm_ray_backend.py * Update cloud_vm_ray_backend.py * format

Fix stream_logs Duplicate Job Handling and TypeError (skypilot-org#…

3acdfd0

…4274) fix: multiple `job_id`

Merge branch 'master' into user_select_policy

80e9a8b

cblmemo reviewed Nov 8, 2024

View reviewed changes

sky/serve/load_balancer.py Outdated Show resolved Hide resolved

cblmemo reviewed Nov 8, 2024

View reviewed changes

sky/serve/service_spec.py Outdated Show resolved Hide resolved

sky/serve/service_spec.py Show resolved Hide resolved

AlexCuadron and others added 4 commits November 7, 2024 22:07

Update sky/serve/load_balancer.py

f7d6c84

Co-authored-by: Tian Xia <cblmemo@gmail.com>

feat(serve): Improve load balancing policy error message and display

0f23771

1. Add available policies to schema validation 2. Show available policies in error message when invalid policy is specified 3. Display load balancing policy in service spec repr when explicitly set

fix(serve): Update load balancing policy schema to match implemented …

6ef89ac

…policies Only 'round_robin' is currently implemented in LoadBalancingPolicy class

linting

c605234

cblmemo reviewed Nov 8, 2024

View reviewed changes

sky/utils/schemas.py Outdated Show resolved Hide resolved

refactor(serve): Remove policy enum from schema

8db4dd9

Move policy validation to code to avoid duplication and make it easier to maintain when adding new policies

AlexCuadron force-pushed the user_select_policy branch from 09f3c41 to 8db4dd9 Compare November 8, 2024 08:53

AlexCuadron added 2 commits November 8, 2024 01:30

fix

b2b7378

linting

96f6418

cblmemo reviewed Nov 9, 2024

View reviewed changes

sky/utils/schemas.py Outdated Show resolved Hide resolved

sky/serve/service_spec.py Outdated Show resolved Hide resolved

AlexCuadron and others added 3 commits November 9, 2024 18:27

Update sky/serve/service_spec.py

3fa28b2

Co-authored-by: Tian Xia <cblmemo@gmail.com>

Fix circular import in schemas.py by moving load_balancing_policies i…

14e244e

…mport inside function

linting

1727596

cblmemo approved these changes Nov 11, 2024

View reviewed changes

cblmemo added this pull request to the merge queue Nov 11, 2024

Merged via the queue into skypilot-org:master with commit 3bfc29e Nov 11, 2024
20 checks passed

cblmemo deleted the user_select_policy branch November 11, 2024 03:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UX - SkyServe] user now is able to select a LB policy from a range of options #4061

[UX - SkyServe] user now is able to select a LB policy from a range of options #4061

AlexCuadron commented Oct 10, 2024 •

edited

Loading

cblmemo left a comment

AlexCuadron commented Oct 17, 2024

cblmemo left a comment

AlexCuadron commented Oct 18, 2024

cblmemo left a comment

AlexCuadron commented Oct 20, 2024

cblmemo left a comment

AlexCuadron commented Oct 21, 2024

cblmemo left a comment

AlexCuadron commented Nov 8, 2024

cblmemo left a comment

AlexCuadron commented Nov 8, 2024

cblmemo left a comment

AlexCuadron commented Nov 10, 2024

cblmemo left a comment

[UX - SkyServe] user now is able to select a LB policy from a range of options #4061

[UX - SkyServe] user now is able to select a LB policy from a range of options #4061

Conversation

AlexCuadron commented Oct 10, 2024 • edited Loading

cblmemo left a comment

Choose a reason for hiding this comment

AlexCuadron commented Oct 17, 2024

cblmemo left a comment

Choose a reason for hiding this comment

AlexCuadron commented Oct 18, 2024

cblmemo left a comment

Choose a reason for hiding this comment

AlexCuadron commented Oct 20, 2024

cblmemo left a comment

Choose a reason for hiding this comment

AlexCuadron commented Oct 21, 2024

cblmemo left a comment

Choose a reason for hiding this comment

AlexCuadron commented Nov 8, 2024

cblmemo left a comment

Choose a reason for hiding this comment

AlexCuadron commented Nov 8, 2024

cblmemo left a comment

Choose a reason for hiding this comment

AlexCuadron commented Nov 10, 2024

cblmemo left a comment

Choose a reason for hiding this comment

AlexCuadron commented Oct 10, 2024 •

edited

Loading