KEDA Scaling not triggering for Kafka topic - Metrics showing 0 messages

[usmansiddiqui6]

We are trying to get a KEDA scaling implemented on our Springboot service that consumes messages from a Kafka topic. We have SSL enabled on the topic and have the following configuration.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-kafka-credentials
  namespace: namespace-dev
spec:
  secretTargetRef:
    - parameter: key
      name: consumer-dev-secret
      key: key
    - parameter: cert
      name: consumer-dev-secret
      key: cert
    - parameter: keyPassword
      name: consumer-dev-secret
      key: keyPassword
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: keda-consumer-dev
  labels:
    app: consumer-dev
  namespace: namespace-dev
spec:
  scaleTargetRef:
    kind: Deployment
    name: consumer-dev
  pollingInterval: 30   # 30 seconds
  cooldownPeriod:  300  # 300 seconds
  minReplicaCount: 2
  maxReplicaCount: 10
  triggers:
    - type: kafka #network feed
      metadata:
        bootstrapServers: corkfk-kafka-0,corkfk-kafka-1,corkfk-kafka-2
        consumerGroup: com__consumer
        topic: com_feed_dev
        lagThreshold: '10' #default is 10
        tls: "enable"
        unsafeSsl: "true"
      authenticationRef:
        name: keda-kafka-credentials

It seems that our metrics are not even picking up the messages are that being pushed to the topic. Is there something we are missing here?

[zroubalik]

Do you see anything problematic in KEDA Operator logs?

[usmansiddiqui6]

Hey @zroubalik , thanks for the reply. Yes we are seeing a repetitive error: "Reconciling ScaledObject".

We applied the keda config via kubectl and saw the following logs:

"Updated HPA according to ScaledObject"
"Initializing Scaling logic according to ScaledObject Specification"

and then the repeating "Reconciling ScaledObject"

[usmansiddiqui6]

@zroubalik Is there anything else I could look further into to figure this out?

[zroubalik]

how many messages do you have in the topice?

@dttung2905 might have an idea?

[usmansiddiqui6]

Our last test we sent 26k messages but we stayed at 1 pod to process them all

My understanding is that if our configuration was incorrect we would get some error from the 'keda-operator' pod but we're just getting "Reconciling ScaledObject" continuously.

[jean-mercier-hivebrite]

i think we have the same behaviour can be due to consumer setting ?

When the pod consumer is up the metric show zero lag keda decide to scale down from 1 to 0
when the pod stop after 5 min of the cooldown period the metric show lag and keda take decision to scale up from 0 to 1

The problem here is that when a pod start seem the metrics is no more accurate return zero.

i already faced similar issue on rabbitmq because the consumer was not limited and take all the message available leading to the metric viewing zero available fix was made by excludeUnacknowledged param and prefetch count on consumer https://www.rabbitmq.com/docs/consumer-prefetch.

Do you now if similar behaviour can happen on kafka ?

seem this can explain also
https://stackoverflow.com/questions/69525142/spring-kafka-consumer-lag-metric-is-always-0

what is the metric keda is looking exactly ?