Fallback is incorrect when using multiple triggers with scalingModifiers

[RRethy]

Report

When using 2 or more triggers and a scalingModifier (e.g. max(trigger_1, trigger_2)), if all triggers fail then the amount that gets scaled to is incorrect.

Expected Behavior

I would expect the fallback to scale to the amount specified in the .spec.fallback.replicas field.

Actual Behavior

It scales to t * n where t is the number of triggers and n is the .spec.fallback.replicas.

Steps to Reproduce the Problem

You can reproduce it with the following minimal manifest:

I also put this into a repo which includes a script to spin up a kind cluster to reproduce this, https://github.com/RRethy/keda-issue-minimum-reproducible.

After applying this manifest, the deployment will eventually get scaled to 24 even though it should scale to 12.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: myscaledobject
  namespace: foobar
spec:
  scaleTargetRef:
    kind: Deployment
    name: mydeployment
  fallback:
    failureThreshold: 4
    replicas: 12
  advanced:
    scalingModifiers:
      formula: "max(trigger_1, trigger_2)"
      metricType: AverageValue
      target: "1"
      activationTarget: "0"
  pollingInterval: 30
  minReplicaCount: 3
  maxReplicaCount: 30
  triggers:
  - type: prometheus
    name: trigger_1
    metricType: AverageValue
    metadata:
      serverAddress: http://fake.svc.cluster.local:9090
      threshold: "1"
      activationThreshold: "0"
      query: >
        max(mymetric1{}[2m])
  - type: prometheus
    name: trigger_2
    metricType: AverageValue
    metadata:
      serverAddress: http://fake.svc.cluster.local:9090
      threshold: "1"
      activationThreshold: "0"
      query: >
        max(mymetric2{}[2m])
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mydeployment
  namespace: foobar
  labels:
    app: busybox
spec:
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox:latest
        ports:
        - containerPort: 8080

Logs from KEDA operator

The logs are long but I put them into the above repo, https://github.com/RRethy/keda-issue-minimum-reproducible/blob/master/keda-operator-logs.txt. There are also logs for the other components in that repo.

KEDA Version

2.12.1

Kubernetes Version

1.27

Platform

Other

Scaler Details

Prometheus

Anything else?

From a glance at the code, it seems to want to ignore the scalingModifiers if there is a fallback active, but this seems like the wrong intention even if it did work. I would expect a scalingModifier with a formula of max(trigger_1, trigger_2) to become max(12, 12) and then 12 (assuming the fallback is 12).

cc. @gauron99

[zroubalik]

@gauron99 is looking at this

[gauron99]

It looks to me like immediately upon deployment the scaled deployment is scaled to fallback value and doesnt "wait" to reach the failureThreshold. Once the threshold is reached the number of replicas is doubled. This has to do with the number of triggers (during my testing I increased the number and the replica number would eventually increase accordingly. It would usually switch between multiples of the fallback value; up to max of number of triggers * fallback value).

On HPA request for metrics (in scaleHandler), the values are all correct as they should be and do not match this bug. However the fallback code is executed for each metric and I think it has to do with parallelization that was introduced, thats the only thing I see that could be the culprit here (i can be wrong ofcourse). Furthermore in scaleExecutor: it looks to me like its scaling to fallback value even before the failureThreshold is reached (this would explain the immediate scale to fallback at the very start).

This is probably related #5359. (and thanks to Jorge for help)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.