Kafka scaler should concurrently query brokers and partitions for their message offsets

[bpinske]

Proposal

Currently, the kafka scaler serially queries kafka brokers for partitions one at a time. For very large topics with many hundreds of partitions, this can be quite slow. During this slow query, the controller-manager hpa loop blocks preventing other HPA actions from taking place.

This has previously been optimized with only querying each broker once for the entire list of all partitions held by the broker, but ultimately this process remains serial.

All brokers should be concurrently queried to minimize time to calculate the total number of unprocessed messages in a topic.

Use-Case

As of now, the kafka scaler is non-performant when multiple kafka-based scaledObjects are querying large topics.

This change would make the kafka scaler usable.

Anything else?

[VerstraeteBert]

Hey @bpinske

I've tried my hand at implementing this. Haven't had the chance to properly test this myself, feel free to try it out :-). Note that I'm quite a golang novice still, especially its concurrency patterns.