Add AWS MSK IAM authentication to Kafka scaler

[adrien-f]

Following the discussion in #5531 and #5540, this adds the support for SASL OAuthBearer with MSK token provider to the Kafka scaler using sarama library.

I've added two new fields, saslTokenProvider (authParams or TriggerMetadata) and awsRegion (TriggerMetadata).

It is currently functional, to make it work, here's an example:

---
apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: default
data:
  sasl: "oauthbearer"
  saslTokenProvider: "aws_msk_iam"
  tls: "enable"
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
  namespace: default
spec:
  secretTargetRef:
  - parameter: sasl
    name: keda-kafka-secrets
    key: sasl
  - parameter: saslTokenProvider
    name: keda-kafka-secrets
    key: saslTokenProvider
  - parameter: tls
    name: keda-kafka-secrets
    key: tls
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: azure-functions-deployment
  pollingInterval: 30
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: localhost:9092
      consumerGroup: my-group       # Make sure that this consumer group name is the same one as the one that is consuming topics
      topic: test-topic
      # Optional
      lagThreshold: "50"
      offsetResetPolicy: latest
      awsRegion: "eu-west-1"
    authenticationRef:
      name: keda-trigger-auth-kafka-credential

I am setting it as a draft while I continue going through the contributing process and pending implementation feedback!

Checklist

• When introducing a new scaler, I agree with the scaling governance policy
• I have verified that my change is according to the deprecations & breaking changes policy
• Tests have been added
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (repo) (if applicable)
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #5540

Relates to #5531

[dttung2905]

Hi @adrien-f , thanks for working on this. On a high level note, it think its good to me. Just a few notes from me.
In my opinion, awsRegion should not be a secret and hence should be in triggerMetadata. Wdyt?

[adrien-f]

Hi @adrien-f , thanks for working on this. On a high level note, it think its good to me. Just a few notes from me. In my opinion, awsRegion should not be a secret and hence should be in triggerMetadata. Wdyt?

Hey @dttung2905, thanks for the review! Indeed, I was not sure where to put awsRegion, to me it's part of the authentication parameters more than the scaler itself, but I guess that's a change we could see later in GetAwsAuthorization and discuss in another issue. I'll move it back to the triggerMetadata.

[zroubalik]

/run-e2e kafka
Update: You can check the progress here

[zroubalik]

This is great! Thanks for the contribution. Could you please also open keda-docs PR to update documentation for this addition.

@JorTurFer @tomkerkhove imho would be great if we can run e2e test against AWS MSK instance

[JorTurFer]

imho would be great if we can run e2e test against AWS MSK instance

This will be more complicated that we'd expect because IIRC, MSK instance has to run mandatory using private networking. This means that we have 2 options:

• deploy an EKS cluster + MSK cluster
• create private networks in both sides and establishing a peering between them. This allows us to use MSK from Azure and IAM is already supported

IDK which is the best option, probably we should check the cost in both cases to not consume more than the budget. I don't have time to do this either atm, but if any contributor is willing to add a PR to testing infra adding the required changes, I can review them

[dttung2905]

@JorTurFer, it is indeed tricky to test , I totally agree

MSK instance has to run mandatory using private networking. This means that we have 2 options:

I just take a peak at the MSK ofifical doc, they do support exposing the broker publicly but you can't make it public at the creation stage

Amazon MSK gives you the option to turn on public access to the brokers of MSK clusters running Apache Kafka 2.6.0 or later versions. For security reasons, you can't turn on public access while creating an MSK cluster. However, you can update an existing cluster to make it publicly accessible. You can also create a new cluster and then update it to make it publicly accessible.

I don't think exposing the bootstrap server endpoint publicly is a good point ( security wise ) and we may still end up with either options as Jorge mentioned above ☝️

[zroubalik]

OK, let's ignore this. Maybe AWS decide to sponsor this in the future :)

[adrien-f]

I will get started on updating the documentation! Thanks a lot for the review 👍

[zroubalik]

I will get started on updating the documentation! Thanks a lot for the review 👍

@adrien-f we plan to do a new release today or tomorrow morning. We can include this change as well, if you are able to provide docs soon.

[zroubalik]

Please add docs and Changelog

[adrien-f]

Hello @zroubalik ! I've pushed kedacore/keda-docs#1381 and a changelog entry

[zroubalik]

/run-e2e kafka
Update: You can check the progress here

[zroubalik]

LGTM