Spot VM Support

[vlerenc]

Stories

• As user/operator I want to use AWS spot, Azure Low-Priotity and/or GCP Preemptible VM instances, so that my landscape runs at lower costs.

Motivation

Money, sure, but also some form of chaos monkey that should help train the application developers that all resources will eventually fail.

Acceptance Criteria

• Machine classes can be specified as having one of these additional "flavors":
  • https://aws.amazon.com/en/ec2/spot/ (requires some field spot bid/price, most likely)
  • https://cloud.google.com/compute/docs/instances/preemptible (seems to be no bidding involved here)
  • https://azure.microsoft.com/en-gb/blog/announcing-public-preview-of-azure-batch-low-priority-vms/ (seems not to be general purpose?)
• Machines are either (at least) labelled or maybe even tainted so that only tolerating pods (with strict replication onto multiple machines) are scheduled onto them

Remarks

Looks like Bosh had the same idea (well, everybody can if they have cattle VMs).

Enhancement/Implementation Proposal (optional)

Ideally, link to EP, e.g. a GEP in Gardener (https://github.com/gardener/gardener/tree/master/docs/proposals), alternatively prose here.

Challenges

• supporting autoscaling for such node-grps. Known issues are:
  • priority expander not working as expected for AWS Spot failure kubernetes/autoscaler#4958
  • Node pool scale up timeout kubernetes/autoscaler#1133 (comment)
• Machine deployment to ignore draining of machines on RECREATE strategy #352

[hardikdr]

GKE enabled support for Pre-emptible VMs: https://cloud.google.com/kubernetes-engine/docs/concepts/preemptible-vm

[vlerenc]

Yes, I saw that quite some time ago. That's why I said in one of our syncs, we won't be the first anymore. It really does make a lot of sense, too. On the other hand, our priorities are right. We know we like to have it eventually, but we can't do everything at the same time.

[vlerenc]

Funny, today I even saw this (thanks @afritzler): https://cloudplatform.googleblog.com/2018/06/Cloud-TPU-now-offers-preemptible-pricing-and-global-availability.html

Maybe we can beat GKE with preemptible TPUs in Kubernetes clusters then? ;-) Just kidding, but TPU support is definitely also interesting and somehow different from how AWS handles GPU support (that already works, because MCM doesn't care, but TPUs must be assigned, @afritzler and @rfranzke told me a couple of days ago).

[vlerenc]

Once we have the time to work on this one (GKE and others support that, too - just saw it with Banzai as well), we might leverage this here: org:banzaicloud repo:spot-price-exporter.

[samedguener]

@vlerenc Are there any updates regarding node pools with hotspot instances? We are looking forward for this!

Best,
Samed

[vlerenc]

No, no update. So far, nobody even contacted us with the concrete need. You are the first. Most workload can't cope with that kind of infrastructure. Can you elaborate about your use case a bit?

cc @hardikdr @prashanth26 @amshuman-kr @juergenschneider

[samedguener]

We are planning to have node pools with hotspot instances

• to run in future batch jobs such as the training of machine learning model (long running batch jobs) on hotspot instances and reschedule them during unavailability. This will allow us to reduce our TCOs on the long-term.
• reduce cost in our development costs by allowing to scheduling pods during PRs tests to such nodepools.

Best,
Samed

[jscarney]

Update on ability to support 'spot' instances across Azure and GCP? Will be useful towards cost savings.

[vlerenc]

There were quite some updates: E.g. AWS, Azure, and GCP now all support spot instances with dynamic prices (Azure and GCP deprecated their old models in favour of the new ones that are all called spot VMs). GCP doesn't support a threshold though, which is less than optimal (you can always look up the price though and act accordingly). Grace periods vary (AWS 120s, Azure and GCP 30s), but all notify and we could use that for immediate drain.

I also looked up auto-scaling groups: Now they all support multiple zones, but only AWS and Azure support mixing on-demand and spot instances. AWS' feature seems strange though, because different than Azure and GCP, the spot price may go beyond even the on-demand price. When the user sets a limit, e.g. at the regular on-demand price, AWS won't add capacity and you are left with the on-demand baseline, but Azure fulfils the request, capped at the on-demand price, so you get your machines still. That's at least how I understood the docs.

Rebalancing is another open point, e.g. never, always, grace_period and/or cost_gap maybe?

[vlerenc]

There are some stakeholders that could still benefit from spot instances as they have enormous fluctuations during the day/week:

[elankath]

Will need to investigate stakeholder clusters to get an idea of their workload behaviour.

[aaronfern]

Closing this issue since it's very old, and there seems to be no traction.

Proper support can only be added only when the MachineDeployment is overhauled to support usage of spot and regular instances. A new issue can be created post that for spot VM support

/close