Pausing keda with either paused-replicas or paused annotation causes the replicas to jump in between 0-1

[rahulkishen18]

Report

I'm using keda to pause deployments in google cloud and we have monitoring with prometheus and grafana. I'm using keda version 2.12, which has two annotations for pause, namely, autoscaling.keda.sh/paused-replicas and autoscaling.keda.sh/paused . According to the docs here, either one can be used to pause a deployment with different use cases.

In practice, I see that when I use either paused-replicas or paused annotation, the replicas bounce around between [1-0] but when I use both together, only then does this "bouncing around" stop and the replicas stay at 0 .
In the grafana plot attached, region A is when paused-replicas is added to the scaled object, region B is when paused is added and the region to the right of B, is when both annotations are added.

Expected Behavior

The expected working of the pause annotations according to the docs here suggest that the annotation autoscaling.keda.sh/paused will pause scaling immediately and use the current instance count while the annotation autoscaling.keda.sh/paused-replicas: "<number>" will scale your current workload to specified amount of replicas and pause autoscaling.

It is also stated that either one of the annotations is typically used based on the scenario and so I’d expect that using either one of the annotations should pause the scaled object successfully and maintain the replicas at 0.

Actual Behavior

Actual behavior as described in the report is that the replicas are only maintained at 0 when both annotations are set together. The region to the right of Region B shows the test when the keda scaled object has both annotations i.e.

metadata:
  annotations:
    autoscaling.keda.sh/paused-replicas: "0"
    autoscaling.keda.sh/paused: "true"

Steps to Reproduce the Problem

1. Setup a deployment in GKE
2. Configure a scaled object to scale this deployment based on a CPU trigger
3. Add either the autoscaling.keda.sh/paused-replicas: "0" or the autoscaling.keda.sh/paused: "true" annotation and check if the replicas remain at 0. This should result in the bouncing around of replicas between 0-1 based on my observations.
4. Now add both the annotations to the metadata to check if the replicas remain at 0. This should force the replicas to 0 based on my observations.

Logs from KEDA operator

No response

KEDA Version

2.12.0

Kubernetes Version

1.27

Platform

Google Cloud

Scaler Details

CPU

Anything else?

No response

[SpiritZhou]

Could you provide more detail about your scaledobject? Did you try it with other scaler like kubernetes-workload? Or you just find it under CPU scaler?

[rahulkishen18]

Could you provide more detail about your scaledobject? Did you try it with other scaler like kubernetes-workload? Or you just find it under CPU scaler?

I just used the CPU scaler - here’s what the scaledobject looks like. I’ve changed the name, namespace and uid for privacy reasons.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  annotations:
    autoscaling.keda.sh/paused: "true"
    autoscaling.keda.sh/paused-replicas: "0"
  creationTimestamp: "2023-10-26T16:30:41Z"
  finalizers:
  - finalizer.keda.sh
  generation: 3
  labels:
    scaledobject.keda.sh/name: dev-barista-gpu-copy
  name: test-deployment
  namespace: test-namespace
  resourceVersion: "1563338046"
  uid: test-uid
spec:
  maxReplicaCount: 40
  minReplicaCount: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test-deployment
  triggers:
  - metadata:
      value: "45"
    metricType: Utilization
    type: cpu

[SpiritZhou]

The CPU/MEM scaler doesn't support minReplicaCount=0, so when the CPU/MEM scaler is created, the target pod will be scaled to 1 immediately. After the annotation is set, the target will be scaled down to 0.

[rahulkishen18]

Yes I agree, but whenever I set the either only the autoscaling.keda.sh/paused or autoscaling.keda.sh/paused-replicas as mentioned in the issue above, I see that the target is not scaled down to 0 and keeps bouncing between 0 and 1. When I set both however, I see that the target remains at 0.

However, the expectation was that using only paused or paused-replicas annotation should have accomplished the same task. Does this make sense ?

[SpiritZhou]

I waited for 5 minutes but did not see the bouncing behavior in my test environment. I only set autoscaling.keda.sh/paused-replicas, and the target pod was scaled to 1 but remained at 0. Do you have any more detailed information that you can provide?

[rahulkishen18]

I checked again and it seems like I had some local issue that caused this to behave weirdly with kube-downscaler that I was using. Thanks for checking! Will close this PR!