Scalers don't retry if connection is lost

[pascallap]

Report

If the scaler rabbitmq is installed or launched while rabbitmq isn't up.
There is no reconnection attempt.

We only get an error:
ERROR scalehandler error resolving auth params {"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledJob", "namespace": "anamespace", "name": "servicename"}, "trigger": 0, "error": "error establishing rabbitmq connection: dial tcp xxx.xxx.xx.xx:5672: i/o timeout"}

And if rabbitmq is available after that there is no reconnection attempt.
The only way to solve this is by restarting the keda-operator.

Expected Behavior

A reconnection retry.

Actual Behavior

No retry.

Steps to Reproduce the Problem

1. Install keda in namespace keda
2. Install rabbitmq in namespace default
3. configure a network policy on default namespace blocking all external traffic.
4. Install a scaler in default namespace
5. ---- Failure --- Error in keda operator log
6. Delete the network policy to allow traffic.
7. ---- Nothing ---- There is no reconnect from keda to rabbitmq.

Logs from KEDA operator

ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledJob", "namespace": "anamespace", "name": "servicename"}, "trigger": 0, "error": "error establishing rabbitmq connection: dial tcp xxx.xxx.xx.xx:5672: i/o timeout"}

KEDA Version

2.5.0

Kubernetes Version

1.19

Platform

Amazon Web Services

Scaler Details

RabbitMQ

Anything else?

This is a big problem, since we are running rabbitmq in cluster.
And we close the clusters each night.

Keda is too fast to restart, and rabbitmq is not properly up when keda tries to establish the connection.

[zroubalik]

Thanks for opening this issue, it seems like that this bug has been introduced by caching mechanism implemented in 2.5.0 #2187 and it is not exclusive to RabbitMQ.

[Lyrositor]

I can confirm this, I have experienced this with the Redis Lists scaler as well.

Reverting to 2.4.0 fixes the issue.

[JorTurFer]

I think that this PR solves the problem