Kafka: Ability to set upper bound to the number of partitions with lag

[DmitrySenin]

Added a way to configure the Kafka scaler to limit the max number of spun-up replicas to the number of partitions with non-zero lag.
This flag is definitely not compatible with allowIdleConsumers. Hence, added the respective validation
Also I don't see much sense in not having a topic when using the new setting. So added a validation check for this case as well

Checklist

• Tests have been added
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update the documentation on (repo) (if applicable)
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #3997

Relates to kedacore/keda-docs#1107

[DmitrySenin]

I would need to some help with testing. Do I need to test all locally? Or there is a test env(with a Kafka scaler) which gets automatically spun up?

[zroubalik]

I would need to some help with testing. Do I need to test all locally? Or there is a test env(with a Kafka scaler) which gets automatically spun up?

There are e2e tests that you can run locally (against k8s cluster) and then we can also trigger them on the PR:
https://github.com/kedacore/keda/blob/main/tests

You should extend this test: https://github.com/kedacore/keda/blob/main/tests/scalers/kafka/kafka_test.go

[zroubalik]

any update on this please?

[DmitrySenin]

I've been busy at work. I should be back with e2e tests by Monday

[zroubalik]

I've been busy at work. I should be back with e2e tests by Monday

gentle ping :)

[zroubalik]

To fix the Static Checks, please add exception rule here: https://github.com/kedacore/keda/blob/main/.golangci.yml#L53

[BojanZelic]

@DmitrySenin any update on this feature? I could give implementing the tests & anything left over a try to get this merged; Would love to see this implemented

[DmitrySenin]

@BojanZelic , I'm unlikely to get back to this one in the next two months. So feel free to pick up.
I think Static Checks need to be fixed and unit tests need to be added

[zroubalik]

Resolved in #5060