KedaScaledJob is unable to scale due to invalid offset with kafka-trigger

[zaheer032]

Report

When we try to produce message on kafka topic we except new keda jobs to be created based on trigger specified(in our case kafka-trigger). However we can see that no jobs are getting created and consumer group also is not created,.

Since lagtreshold is specified as "1" and kept minreplicacount as "0" : one-job should be created whenever there is message on Kafka queue and should scale according to lagthreshold.
To add to the behaviour: there is no existing consumer group. When we manually add consumer group it starts working

Expected Behavior

Keda jobs should scale based on kafka-trigger

Actual Behavior

Keda consumer jobs should be created

Steps to Reproduce the Problem

1. Create the kafka-scaledjob

{{- if .Values.ServiceTagsMessageConsumer.enabled -}}
{{ if .Values.keda.enabled -}}
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: {{ .Release.Name }}-tags-message-consumer
  namespace: {{ .Release.Namespace }}
spec:
  jobTargetRef:
    parallelism: 1
    template:
      spec:
        containers:
          - name: {{ .Values.assetServiceTagsMessageConsumer.name }}
            image: {{ .Values.image.repository }}
            args:
              - python
              - manage.py
              - tags_consumer
            imagePullPolicy: {{ .Values.image.pullPolicy }}
            env:
              - name: SSL_CA_LOCATION
                value: {{ .Values.kafka.sslCaDirMount }}/ca.crt
              - name: SSL_CA_DIR_MOUNT
                value: {{ .Values.kafka.sslCaDirMount }}
              - name: SSL_SECRET_NAME
                value: {{ .Values.kafka.sslSecretName }}
              - name: SSL_USER_KEY_LOCATION
                value: {{ .Values.kafka.sslUserCertDir }}/user.key
              - name: SSL_USER_CERT_LOCATION
                value: {{ .Values.kafka.sslUserCertDir }}/user.crt
              - name: SSL_USER_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: {{ .Release.Name }}-kafka-user-certs
                    key: user.password
              - name: SSL_USER_MOUNT_PATH
                value: {{ .Values.kafka.sslUserCertDir }}
              - name: ASSET_TAGS_CONSUMER_GROUP_ID
                value: {{ .Values.kafka.ASSET_TAGS_CONSUMER_GROUP_ID }}
              - name: KAFKA_ASSET_TAG_TOPIC
                value: {{ .Values.tags_consumer_service.consumption_topic }}
              {{- include "env" . | nindent 14 }}
            resources:
              {{- toYaml .Values.assetServiceTagsMessageConsumer.resources | nindent 14 }}
            volumeMounts:
              - name: user-certs
                mountPath: {{ .Values.kafka.sslUserCertDir }}
              - name: ca-certs
                mountPath: {{ .Values.kafka.sslCaDirMount }}
        volumes:
          - name: user-certs
            secret:
              secretName: {{ .Release.Name }}-kafka-user-certs
          - name: ca-certs
            secret:
              secretName: kafka-cluster-ca-cert
  pollingInterval: 60
  maxReplicaCount: 100
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 5
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: "kafka-kafka-bootstrap.kafka:9093"
        consumerGroup: "AssetTagConsumer"
        topic: {{ .Values.tags_consumer_service.consumption_topic }}
        lagThreshold: "1"
        offsetResetPolicy: latest
      authenticationRef:
        name: {{ .Release.Name }}-keda-trigger-auth-kafka-credential
  {{- end }}
{{- end }}

2. Produce a message on kafka-topic

3. Logs for Keda operator

Logs from KEDA operator

KEDA Version

2.6.1

Kubernetes Version

1.23

Platform

Amazon Web Services

Scaler Details

kafka-trigger

Anything else?

No response

[JorTurFer]

Hi @zaheer032
I'm not an expert on kafka tbh, but I can see in KEDA logs that there are some errors related with any wrong configuration.
Invalid offset found .....

Could you check that whole message and verify that there is not any misconfiguration in KEDA or Kafka?

[zroubalik]

Please check this related issues: #2612 #2033

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[tomkerkhove]

Please check this related issues: #2612 #2033

Closing as issue is stale and above mentioned issues are closed