Support GCP

[deliahu]

Notes

• Update some docs (e.g. how to install GPUs)

[amutz]

When GCP support is added, is there any intention of supporting deploying models to Google Cloud Run?

It is their hosted Knative and is dead simple serverless containers. We use GCR to evaluate predictions in production and are extremely happy with it.

[ospillinger]

Hey Andrew, thanks for your feedback! I agree that Knative and GCR are super relevant and we just spent some time evaluating them. That being said, we built Cortex on top of Kubernetes to simplify supporting all cloud providers so our current plan is to run on GKE but that could definitely change. Our technology decisions are primarily focused on optimizing developer experience and minimizing the cost for running inference at scale so we try to avoid managed services that cost more as usage increases. We also try to abstract the underlying infrastructure services to allow users to focus on ML, so ideally whether we run on GKE or GCR shouldn't matter too much. Are there any features in GCR that you think Cortex is missing?

[amutz]

With Cortex on GKE will I still have to reason about capacity? Selecting number of nodes for peak load? Will I be able to scale to zero nodes when there is no load?

[ospillinger]

Cortex doesn't run on GKE yet, I'm sorry if I mislead you. The latest version of Cortex can autoscale based on load up to the maximum number of nodes that you set. Our next release will include the ability to autoscale the inference nodes to 0 if no deployments are running though there will always be a Cortex operator (server) node unless you spin down the entire cluster. The operator resource requests are small so an inexpensive instance is sufficient. When we add GCP support, all this functionality will be ported over.

[sam-writer]

Is it possible to know which tickets need to be done to support GKE? Or does that work still need to be identified?

[ospillinger]

Hey @sam-qordoba, we have some initial thoughts, but we have not fully scoped it out yet. I'd be happy to jump on a call to discuss if you'd like, feel free to email me at omer@cortexlabs.com

[gautamchitnis]

I am interested in helping cortex support GCP as I am currently planning to use GCP for my product deployment. Do let me know how I can contribute.

[ospillinger]

Hi @gautamchitnis, thank you for offering to contribute! We still don't know exactly when we'll have time to work on GCP support. If you're interested in contributing in other ways please feel free to email me at omer@cortexlabs.com and we can set up some time to chat.