[Core] SkyPilot Client-Server Architecture

[Michaelvll]

This PR rearchitects SkyPilot to be a client-server architecture, i.e. separating the backend from the frontend, so that the
backend can be separately deployed.

What is new?

1. A way to deploy SkyPilot for an organization to have a single pane of glass view for the resources.
2. Multi-tenant SkyPilot allowing multiple users to collaborate (see everyone's resources, share resources, etc)
3. Asynchronous CLI/Python SDK
4. A new set of RESTful APIs and API server-related interfaces

Architecture

Behaviors

Local API server (individual users)

Without deploying a remote API server, SkyPilot still acts normally, with a local API server automatically started.

Remote API server (multi-user organizations)

A user can now deploy a remote API server and have the local client connect to that API server, so that multiple clients can connect to the same API server, i.e. having a single pane of glass of the resources used by multiple users/clients.

Disruptive Changes

1. The SkyPilot Python SDK are now asynchronous by default, i.e. it returns a future, request_id that you can wait or stream with sky.get(request_id) and sky.stream_and_get(request_id)
2. Storage objects no longer eagerly upload files during creation, but waits until a cluster/job is launched.

For more user facing details, see API server docs hosted here: https://docs.skypilot.co/en/client-server/reference/api-server/api-server.html

For more developer facing details, see the readme:

Tests

Local API Server

• pytest tests/test_smoke.py --aws. Fail on [Test] Azure storage test fails probably due to wrong handling of the storage account in the test #4672, which should be fine for now.
• pytest tests/test_smoke.py --gcp
• pytest tests/test_smoke.py --kubernetes
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

Remote API Server

An API server deployed on GKE cluster, deployed with: https://docs.skypilot.co/en/client-server/reference/api-server/api-server-admin-deploy.html

• pytest tests/test_smoke.py --aws
• pytest tests/test_smoke.py --gcp
• pytest tests/test_smoke.py --kubernetes
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

TODO

• Add more tests above

Acknowledgement

We thank all our users for providing valuable feedback during the alpha/beta testing, and all contributors who made this PR possible: @romilbhardwaj @concretevitamin @cg505 @KeplerC @zpoint @yika-luo @weih1121.

[Michaelvll]

/smoke-test --aws

[cg505]

do we use this??

[Michaelvll]

Good catch! There are some merging conflicts. I updated it now.

[zpoint]

/smoke-test --aws -k test_env_check

[zpoint]

/smoke-test --aws -k test_env_check

[zpoint]

/smoke-test --managed-jobs

[Michaelvll]

Good catch! There are some merging conflicts. I updated it now.

[Michaelvll]

@cg505, is this still the case? Do we still rely on the same ray job ID and managed job ID?

[cg505]

Not sure. I think no but there might still be some place (e.g. in log streaming code) that assumes it. Probably best to leave this in.

[Michaelvll]

@cg505, I added this so that jobs failed before scheduler is triggered for the job will be marked as FAILED_CONTROLLER. Wdyt?

[cg505]

A bit concerned here, since the job_lib status could be failed but the managed job was successfully submitted, then something else failed after. In that case we could race with the scheduler and get into some weird undefined behavior.
If we want to add this check, we should also verify that the schedule_state is INACTIVE (submit to scheduler failed), and we need to check schedule_state after we check the job_id status. Otherwise, if the job is WAITING or some other schedule_state, we need to make sure to stop the controller process.

[Michaelvll]

I see, I can add an additional condition for checking the schedule_state to be INACTIVE. I was assuming the job_lib status should always be SUCCEEDED if the managed job was successfully submited.

[cg505]

There is also a question of what we want to do if the job_lib status is some FAILED state but the job was successfully submitted (schedule_state is not INACTIVE). This seems like a problem, so FAILED_CONTROLLER is reasonable. We just have to be careful about how to do it:

• set job to DONE
• get controller process PID
• if not None, kill controller process and children
• clean up cluster

or something like that

[Michaelvll]

I am not sure in what case it could happen that the job_lib can fail when the job is current submitted to scheduler, since the job controller's run section seems only inlcudes the submission to scheduler?

[cg505]

e.g. some ray issue out of our control

[cg505]

Or even in the submission, set_waiting succeeds but maybe_schedule_next_jobs crashes for some reason

skypilot/sky/jobs/scheduler.py

Lines 186 to 188 in 54fe787

	with filelock.FileLock(_get_lock_path()):
	state.scheduler_set_waiting(job_id, dag_yaml_path)
	maybe_schedule_next_jobs()

[cg505]

we need to double check the schedule_state after job_lib.get_status to avoid a race

[Michaelvll]

Hmm, good point! I tried to add the schedule state check again here, and the logic becomes quite complicated. I now revert it to only check the FAILED_SETUP case, which should be safe to only check the job_lib state. PTAL.