Provide support for explicitly pausing autoscaling of ScaledObjects at the current replica count

[tomkerkhove]

Proposal

Similarly to #3303, it would be nice to provide support for explicitly pausing autoscaling of ScaledObjects at the current replica count so that it no longer scales out/in.

This should be done by applying the autoscaling.keda.sh/is-paused: true annotation on the ScaledObjects resource. When this is the case, the ScaledObject just sticks with the current amount of replicas when it was applied.

Use-Case

Pause autoscaling without having to check how many replicas we run today.

Anything else?

This is in addition to our existing autoscaling.keda.sh/paused-replicas: <replica-amount> which allows you to specify a replica count.

When both annotations are applied, the following logic should be used:

• If autoscaling.keda.sh/paused-replicas is specified, stop autoscaling and puts to given amount
• Otherwise, if autoscaling.keda.sh/is-paused is there, just stop autoscaling and keep it at the current instance count
• Otherwise, keep scaling 🚀🚀🚀

As a workaround, people can use autoscaling.keda.sh/paused-replicas as following:

replicas=$(kubectl get deploy <deployment>  -n <NS> -o=jsonpath='{.status.replicas}') | kubectl annotate scaledObject -n <NS> <scaledObject name>  "autoscaling.keda.sh/paused-replicas=$replicas" 

Follow-up for #944

[jbilliau-rcd]

When the annotation is set to is-paused, is the goal that the user would be able to manually control their deployment replicas at any number and autoscaling as a whole is paused, and won't overwrite the users changes? Or is it "paused" at that specific, current replica count, and any different number will be snapped back? If the former, that would be great; we have users who want to be able to, during troubleshooting or load tests, manually control their replicaCount and turn off autoscaling. Will this change accomplish that?

[tomkerkhove]

When the annotation is set to is-paused, is the goal that the user would be able to manually control their deployment replicas at any number and autoscaling as a whole is paused, and won't overwrite the users changes?

This will be a side-effect, but should not happen actually. Instead, end-users should apply the autoscaling.keda.sh/paused-replicas: <replica-amount> annotation on the SO/SJ. That way the autoscaling rules don't change nor the effective deploy target as KEDA manages the scaling on that.

So once you apply that annotation, KEDA will scale the target workload to the specified instance count (implicitly this will change your deployment in this case) and stop scaling it until the annotation is removed.

Hope that clarifies things?

[jbilliau-rcd]

Yeah, would be nicer if it just paused autoscaling entirely, basically pretending like the HPA object doesnt exist, effectively giving control back to the user. I can already achieve what autoscaling.keda.sh/paused-replicas: <replica-amount> will do; just set the min and max to the same number. That effectively "pauses" scaling since, well, they are the same number :)

[tomkerkhove]

If you just want to pause in current replica count, you can already do that today: https://keda.sh/docs/latest/concepts/scaling-deployments/#pause-autoscaling

[jbilliau-rcd]

hmmm maybe we have different definitions of what I'm trying to say, haha. I just tried that, and by setting my ScaledObject with that annotation, autoscaling.keda.sh/paused-replicas: "0", it scaled my deployment down to 0 (was running 2 replicas). If I try setting my deployment replicas to 3, they instantly get terminated as soon as they spawn.

It seems the pause functionality still gives Keda control....I want to take control back and modify the deployment on a whim, basically as if no ScaledObject or HorizontolPodAutoscaler object existed. Currently, and with this PR, the control still rests in the hands of the ScaledObject, and thus, Keda.

[tomkerkhove]

autoscaling.keda.sh/paused-replicas is not implemented yet so I don't know what is doing that :)

Or am I missing something @JorTurFer @zroubalik?

What PR are you referring to @jbilliau-rcd?

[jbilliau-rcd]

@tomkerkhove what do you mean it's not implemented yet? It's in the official docs! https://keda.sh/docs/2.8/concepts/scaling-deployments/#pause-autoscaling

It was implemented in #944

[tomkerkhove]

Oh yes, I have it the other way around - Sorry! Would you mind opening a separate bug for this please? This feature is to allow autoscaling.keda.sh/is-paused: true which just stops at the current instance count.

[jbilliau-rcd]

Right, that's fine, just wanted to confirm that's what it did and not just pause scaling entirely, regardless of manually modified deployment replica count. We good!

[tomkerkhove]

Well just for sake of clarity, if you apply autoscaling.keda.sh/paused-replicas: 10 and manually change the deployment KEDA should scale back to 10.

The goal of autoscaling.keda.sh/is-paused: true is basically saying stop scaling, regardless of how many instances we have now.

[ChayanBansal]

For clarity on the expected scenario:
Case 1: When is-paused is set true when there are already some replicas present - don't scale even if the user keeps on changing minReplicas, maxReplicas or paused-replicas.
Case 2: What if the is-paused is set to true at the start itself? Is the expected behavior to not spawn any instance, i.e., treat it as if the minReplica and maxReplica count is set to 0?

PS: I am asking this since I am planning to work on this as my first contribution @tomkerkhove

[tomkerkhove]

The gist is that KEDA should not perform any scaling when is-paused is applied and avoid it from doing so (their custom or HPA logic)

For clarity on the expected scenario: Case 1: When is-paused is set true when there are already some replicas present - don't scale even if the user keeps on changing minReplicas, maxReplicas or paused-replicas.

Paused replicas is not in the CRD so this is not applicable, otherwise - Yes.

If they add the paused-replicas annotation then that one takes over and the specified amount must be respected.

Case 2: What if the is-paused is set to true at the start itself? Is the expected behavior to not spawn any instance, i.e., treat it as if the minReplica and maxReplica count is set to 0?

If the attribute is applied then simply don't do anything and make sure that the scale target just runs whatever it runs.

PS: I am asking this since I am planning to work on this as my first contribution @tomkerkhove

Awesome, thank you! Can you make sure that the CRD events also has entries on when it gets paused and unpasued please?

[pintu0946]

HI Everyone
I'M working with keda version 2.12.0, But i try to apply my consumber deployment file so find this i know this is API version mismatch error . Could you please Support me which version will be support.

(base) pc@lenvo:~/Documents/scw-heb$ kubectl apply -f consumber.deployment.svc.yaml
resource mapping not found for name: "rabbitmq-consumer" namespace: "default" from "consumber.deployment.svc.yaml": no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1"
ensure CRDs are installed first
resource mapping not found for name: "rabbitmq-consumer-trigger" namespace: "default" from "consumber.deployment.svc.yaml": no matches for kind "TriggerAuthentication" in version "keda.k8s.io/v1alpha1"
ensure CRDs are installed first

[zroubalik]

@pintu0946 the apiVersion is keda.sh/v1alpha1 not keda.k8s.io/v1alpha1

[pintu0946]

Thanks for Support,