feat: Add a Prometheus metric for measuring the scalable object loop processing deviation

[JorTurFer]

Checklist

• Tests have been added
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update the documentation on docs(operate): add keda_internal_scale_loop_latency metric keda-docs#1165
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #4702

[github-actions]

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

While you are waiting, make sure to:

• Add an entry in our changelog in alphabetical order and link related issue
• Update the documentation, if needed
• Add unit & e2e tests for your changes
• GitHub checks are passing
• Is the DCO check failing? Here is how you can fix DCO issues

Learn more about:

• Our contribution guide

[JorTurFer]

/run-e2e sequential
Update: You can check the progress here

[zroubalik]

I wonder, do you see any actualy latency that could be mesured and exposed here? I mean one in the loop itself? We are already reporting latency per scaler. I am not sure if this kind of metric won't be confusion for users?

[JorTurFer]

I wonder, do you see any actualy latency that could be mesured and exposed here? I mean one in the loop itself? We are already reporting latency per scaler. I am not sure if this kind of metric won't be confusion for users?

The difference here is what are we measuring. In one case, we are measuring the trigger latency and in the other case, we are measuring the deviation between loops. Both are similar, but they are different. For example:
If I have a single ScaledObject with pollingInterval: 1 and I have 12 triggers with 100ms of latency, the latency looks nice, but the ScaledObject will have a deviation of 200 ms all the time.

These new metrics measure the difference between the expected execution time and the real execution time, so we are getting a real picture of the overload that we have, if we should have been executed the loop at X time and we execute it later, it's because something is slowing us, it could be due to the triggers' latency but also due to an overload.

If we have throttling, both metrics will increase, but the increase of keda_scaler_metrics_latency doesn't mean an overload in KEDA (because upstream can be responding slowly) and these new metrics do mean it

[JorTurFer]

For example, this is how it looks when there is an overload (I have forced it using 1000 SO with pollingInterval:1 and cpu: 120m) and upstream without any problem:

This is how it looks when KEDA isn't overloaded (cpu: 1) and it's the upstream who responses slowly:

[zroubalik]

The numbers are interesting, great stuff.

Let's procced with keda_internal_scale_loop_latency as we discussed offline.

[JorTurFer]

/run-e2e sequential
Update: You can check the progress here

[zroubalik]

LGTM, great stuff!

[zroubalik]

/run-e2e sequential
Update: You can check the progress here

[tomkerkhove]

Let's make sure we also update our documentation to ensure everyone knows what it represents and how to interpret it

[JorTurFer]

Let's make sure we also update our documentation to ensure everyone knows what it represents and how to interpret it

yeah, I plan to do it later on, I just did this PR before to ensure that we include it in the release (if the release would have been done today).
Docs update incoming