[Spot] Make the controller resources configurable #2040
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Michaelvll
changed the title
Michaelvll
commented
Jun 6, 2023
sky/execution.py
Outdated
Comment on lines
689
to
693
| # Backward compatibility: if the user changed the spot-controller.yaml.j2 | ||
| # to customize the controller resources, we should use it. | ||
| controller_task_resources = list(controller_task.resources)[0] | ||
| if not controller_task_resources.is_same_resources(sky.Resources()): | ||
| controller_resources = controller_task_resources |
There was a problem hiding this comment.
These may not be necessary, but some of our users should have changed the spot-controller.yaml.j2 to make their jobs work.
There was a problem hiding this comment.
Thanks @Michaelvll, mostly looks good to me.
Q: what's the error shown (if any) if a user has a stopped controller on cloud X, and now spot launch with a different resources in the config?
| @@ -12,3 +12,5 @@ | |||
| SPOT_FM_FILE_ONLY_BUCKET_NAME = 'skypilot-filemounts-files-{username}-{id}' | |||
| SPOT_FM_LOCAL_TMP_DIR = 'skypilot-filemounts-files-{id}' | |||
| SPOT_FM_REMOTE_TMP_DIR = '/tmp/sky-spot-filemounts-files' | |||
|
|
|||
| CONTROLLER_RESOURCES = {'disk_size': 50} | |||
Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>
Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>
…into config-controller
Michaelvll
commented
Jun 6, 2023
There was a problem hiding this comment.
Thanks for the review @concretevitamin!
If the user manually change the cloud for the spot controller, when the controller exists on another cloud. The following error will be shown:
> sky spot launch -n test2 echo hi
Task from command: echo hi
Launching a new spot task 'test2'. Proceed? [Y/n]:
Launching managed spot job test2 from spot controller...
Launching spot controller...
sky.exceptions.ResourcesMismatchError: Requested resources do not match the existing cluster.
Requested: 1x AWS(cpus=4+, disk_size=50)
Existing: 1x GCP(n2-standard-4, disk_size=50)
To fix: specify a new cluster name, or down the existing cluster first: sky down sky-spot-controller-9ce1ce58
I think it should be fine to show that. Wdyt?
LGTM, we can see if users have any feedback. |
concretevitamin
approved these changes
Jun 7, 2023
Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>
Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closing #2007, #1974, #1078.
It also mitigates #1094, as the user now can specify more CPUs for the controller.
With this PR, the user can specify resources spec for the spot controller so as to change the default cloud, customize the disk size or change the number of CPU cores (the maximum number of spot jobs == 2x #CPUs).
To customize the spot controller resources, the user can have a
~/.sky/config.yamlwith the following fields:Tested (run the relevant ones):
sky spot launch -n test echo hiwith the~/.sky/skypilot_config.yamlabove.sky spot launch -n test2 echo hirun again to test the availability to launch a second job.pytest tests/test_smoke.pypytest tests/test_smoke.py::test_fill_in_the_namebash tests/backward_comaptibility_tests.sh