Allow cluster-autoscaler to run on spot if nothing else is available

[johngmyers]

An alternative to #14591

Fixes #14411

[olemarkus]

This is not a solution for spot-only clusters, so I don't think it safely fixes #14411

[johngmyers]

@olemarkus could you expand on why this isn't a solution for spot-only clusters?

[olemarkus]

For spot-only, the only really safe place to put CAS would be on the control plane. However, using NTH in addition would probably mitigate this to the extent it is a very unlikely scenario.

If you have many IGs with very narrow mix of insistence types, you can still run into this though, but CAS trying to scale up an IG that has no spot capacity, and then get evicted before it has the chance to jump to the next one.

But yeah, this all may be theoretical since we also run two CAS instances, and the likelyhood of both getting reaped is in itself fairly low.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: olemarkus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [olemarkus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johngmyers]

Cluster Autoscaler is system-cluster-critical, so there's a good chance of it rescheduling if it gets evicted.