unable to establish sasl connection to aws msk cluster

[QualoZe0t]

Report

Hi,
i am trying to deploy scaledObject using by sasl but getting errors in keda-operator (see below). I am not aware of wrong yaml or atleast unable to find error as connection using port 9092(plaintext) was working without any issue

Expected Behavior

ScaledObject have to be created properly

Actual Behavior

	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-01-11T10:44:41Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-kafka-scaledobject","namespace":"data"}, "namespace": "data", "name": "keda-kafka-scaledobject", "reconcileID": "c6b5b3c3-2469-49a6-a27f-c37492f0bf31", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: unexpected EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234

Steps to Reproduce the Problem

1. created secret

apiVersion: v1
kind: Secret
metadata:
  name: keda-msk-data
  namespace: data
data:
  sasl: "scram_sha512"
  username: "test"
  password: "test"

TLS= is not defined so by default should be disabled
2. created triggerauth

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
  namespace: data
spec:
  secretTargetRef:
  - parameter: username
    name: keda-msk-data
    key: username
  - parameter: password
    name: keda-msk-data
    key: password
  - parameter: sasl
    name: keda-msk-data
    key: sasl

3. created scaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: keda-kafka-scaledobject
  namespace: data
spec:
  scaleTargetRef:
    name: kafka-test
  pollingInterval: 30
  minReplicaCount: 1
  maxReplicaCount: 5
  advanced:
    restoreToOriginalReplicaCount: true
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka-cluster:9096
      consumerGroup: kinesis-adapter
      topic: lhe-lake-party-fact-party
      lagThreshold: "2"
      offsetResetPolicy: latest
    authenticationRef:
      name: keda-trigger-auth-kafka-credential

Logs from KEDA operator

	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-01-11T10:44:41Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-kafka-scaledobject","namespace":"data"}, "namespace": "data", "name": "keda-kafka-scaledobject", "reconcileID": "c6b5b3c3-2469-49a6-a27f-c37492f0bf31", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: unexpected EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234

KEDA Version

2.8.1

Kubernetes Version

< 1.23

Platform

Amazon Web Services

Scaler Details

Kafka

Anything else?

No response

[QualoZe0t]

I have tried to configure secrets + triggerauth with tls=enabled/disabled but still same error. Things is that if I am using keda app version 2.8.1 which sarama client is used? based on the compatibility https://github.com/Shopify/sarama/blob/main/CHANGELOG.md version 1.29 (release 2021) should support kafka version 2.8.0. We are using tls on kafka as this is requirement as sasl is enabled also Plaintext is completely off . TLS with configuration ACM is disabled

[dszabome]

I'm also interested in the resolution of this issue. Thanks @QualoZe0t to bring this up.

[JorTurFer]

Does sarama support aws msk cluster or maybe an aws client is needed? If sarama supports it, are you willing to crontibute with needed changes?

[joaopuccini]

i am using Keda app version 2.7.1 and kafka version 2.7.0 and getting same issue once AWS MSK is using SASL SCRAM.

i test with sasl: none, plain_text, scram_sha512 and no results, always the same error.

2023-01-27T10:15:22-03:00 1.6748253225245714e+09	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject",  "error": "error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

i try also, without TriggerAuthentication and Secret, the error was the same.

[dttung2905]

@QualoZe0t I think one possible reason could be pure VPC network problem. Were you able to reach from the same machine to MSK cluser using a different listener ( plaintext listener for example ) ?

@JorTurFer Another direction I have been looking into: from the official MSK doc, we need to have a truststore as well. May be we need to set InsecureSkipVerify to false in KEDA sarama config, to skip server side cert authentication ( source ). I can help with a fix attempt but I don't think I'm able to test with an AWS MSK cluster 😄

[JorTurFer]

@JorTurFer Another direction I have been looking into: from the official MSK doc, we need to have a truststore as well. May be we need to set InsecureSkipVerify to false in KEDA sarama config, to skip server side cert authentication ( source ). I can help with a fix attempt but I don't think I'm able to test with an AWS MSK cluster

I was afraid about this… AWS CA is not trusted on Linux, which could be a problem :(
IMO we should add a mechanism to trust on custom provided CAs, something like a specific path that KEDA checks during the startup and register all of them certs there in the system as trusted CA. @zroubalik WDYT?

[QualoZe0t]

@dttung2905 I was able to confugure KEDA if aws msk has enabled plain text. Guys there are two configurations:

1. enabling ACM -> then private key is inevitable
2. without ACM -> so then you dont need key as ACM is provate CA is disabled

tls setup in aws:
TLS client authentication through AWS Certificate Manager (ACM): NOT enabled
AWS Private Certificate Authority (CA) --> not defined completely disabled
Encryption options for data in transit within cluster and between clients and cluster:

• between brokers: enabled by default
• between client and broker: enabled by default

[JorTurFer]

I have created an issue to track the addition of the custom CA support #4168

[JorTurFer]

is anyone using KEDA from main? The custom CA support feature is already merged and would be nice if anyone (who is using KEDA from main) checks if after adding AWS CA to KEDA pods it works or not

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.