Multiple Triggeres for a scaledobject not working as expected.

[KasyapG]

Report

I am trying to implement schedule-based scaling using keda where trying make the pods to 0 during weekends and faced a 2-part issue.

1. Noticed that cron job is only scaling up and down during the scheduled window. For this I tried to make the pods to zero initally and scale up during the weekdays.
2. However, I wanted to add another trigger to make sure if any traffic comes in it does not encounter error due to pod being 0. The multiple trigger is not working as expected when applied both cron and the kafka triggers both are not working.

Below is the scaledobject used.

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: 60486a51-7069-4d45-d1be-a0069c86f864
data:
  authMode: "c2FzbF9wbGFpbnRleHQ="
  username: "bTMyMjk1QG11bGVzdmMuYXR0LmNvbQ=="         
  password: "*****************"                        # masked here
  tls : "ZGlzYWJsZQ=="
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
  namespace: 60486a51-7069-4d45-d1be-a0069c86f864
spec:
  secretTargetRef:
  - parameter: authMode
    name: keda-kafka-secrets
    key: authMode
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: keda-kafka-queue2
  namespace: 60486a51-7069-4d45-d1be-a0069c86f864
spec:
  scaleTargetRef:
    name: keda-kafka-queue2
  pollingInterval: 3
  #cooldownPeriod:  30                               # Optional. Default: 300 seconds
  minReplicaCount: 0   #Optional Default 0
  maxReplicaCount: 10
  fallback:                                          # Optional. Section to specify fallback options
    failureThreshold: 4                              # Mandatory if fallback section is included
    replicas: 6
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: "cl001.3pc.com:9097"
      #insecure-skip-tls-verify: "true"
      consumerGroup: consumerGroup.clm.dev.keda       # Make sure that this consumer group name is the same one as the one that is consuming topics
      topic: mulesvc.clm.dev.keda
      # Optional
      lagThreshold: '1'
      #activationLagThreshold : '1'
    authenticationRef:
      name: keda-trigger-auth-kafka-credential
  - type: cron                                        #Testing for 30 mins
    metadata: 
      timezone: Asia/Kolkata
      start: 30 15 * * *                    
      end: 00 16 * * *
      desiredReplicas: "4"
    authenticationRef:
      name: keda-trigger-auth-kafka-credential

Also, the scaled object when applied only shows the first trigger (Not showing and functionality is also not working). used
kubectl get scaledobject -A.

Could someone please look into this and suggest if the multiple trigger feature is working in keda if so am I using it incorrectly. Please suggest.

Expected Behavior

• As there are 2 triggers the scaling up to 4 should happen during the 3:30 to 4 window.
• And when api is triggered with load from post man the pods should start spinning up.

Actual Behavior

• The scaling to desired replicas is not happening, neither the scaling up when traffic is received is working.

Steps to Reproduce the Problem

1. Yaml given above can used and applied on any api deployed.
2. once applied successfully wait for the duratim to see pods spinning or not from 0.

Logs from KEDA operator

example

KEDA Version

None

Kubernetes Version

None

Platform

None

Scaler Details

No response

Anything else?

No response

[JorTurFer]

Hello,
Could you share KEDA operator logs? it should work as you've described. It does a MAX between all the metrics

[KasyapG]

I noticed the logs had issue with the kafka connector, so replaced the kafka trigger with memory trigger to test.

spec:
scaleTargetRef:
name: keda-kafka-queue2
pollingInterval: 1.
#cooldownPeriod: 30 # Optional. Default: 300 seconds
minReplicaCount: 0 #Optional Default 0
maxReplicaCount: 10
triggers:

• type: cron
  metadata:
  timezone: Asia/Kolkata
  start: 05 14 * * *
  end: 10 14 * * *
  desiredReplicas: "4"
  authenticationRef:
  name: keda-trigger-auth-kafka-credential
• type: cpu
  metadata:
  type: AverageValue
  value: "70"
  authenticationRef:

I see cron wokring as expected, however the scaledobjects show only cron.

Also, after cooling down it is not going back to 0 replicas instead scaling down to 1 from 4.

My requirement is to scale down the pods to 0 during the weekend, however using cron jobs I was able to only scale up and not scale down.

Could you please suggest an alternative.

[JorTurFer]

If you have CPU and/or Memory + other triggers, you can scale to 0 based on those other triggers and the CPU/memory won't block the scale to zero (since v2.11).

In your case, I guess that you have multiple scalers (cron + kafka) and all of them can scale from/to 0, so if there is at least one that requires scaling from 0, you will have at least 1 instance and this is because as default, HPA controller does a MAX between all the metrics and KEDA respects that approach (even cron isn't active, kafka is, so KEDA scalers to 1 your workload).

In this case, we recently (v2.12) added a new experimental formula for modifying the scalers applying custom formulas. This allows you for enabling scenarios like setting cron desiredReplicas : 1 and generating the calculated value as Kafka * Cron, so during cron active period, you will have Kafka required instances (lag * 1 = 1) and outside active period, you will have 0 (lag * 0 = 0).

In your case, your ScaledObject could look like:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: keda-kafka-queue2
  namespace: 60486a51-7069-4d45-d1be-a0069c86f864
spec:
  ....
  advanced:
    scalingModifiers:
      formula: "kafka_scaler * cron_scaler"
      target: "1"
  triggers:
  - type: kafka
    metadata:
      ...
      # lagThreshold: '1' THIS VALUE ISN'T REQUIRED AS IT'S OVERRIDED
    authenticationRef:
      name: keda-trigger-auth-kafka-credential
    name: kafka_scaler
  - type: cron                                        #Testing for 30 mins
    metadata: 
      ...
      desiredReplicas: "1" # THIS IS REQUIRED AS IT'S RETURNED VALUE
    name: cron_scaler

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.