Incorrect calculation of metric from Prometheus

[BorisDr]

Report

To scale deployment based on pods qty in namespace I am using this query to Prometheus:
threshold: '10' query: sum(kube_pod_status_phase{namespace="namespace",phase="Running"}) / sum(kube_pod_status_phase{namespace="namespace",phase="Running",pod=~"workload2.*"})
While direct query to Prometheus show number 15 command kubectl describe hpa shows: "prometheus-http---prometheus-server-prometheus-svc-srv-rps" (target average value): 7 / 10

Expected Behavior

Metrics in keda hpa should be the same as in Prometheus

Actual Behavior

Metrics in keda hpa are not the same as in Prometheus

Steps to Reproduce the Problem

1. deploy 2 workloads in any namespace. One for reference, second to scale based on 1-st workload qty.
2. scale workload-1 up to 10 replicas
3. create keda ScaledObject with Prometheus query like sum(kube_pod_status_phase{namespace="namespace",phase="Running"}) / sum(kube_pod_status_phase{namespace="namespace",phase="Running",pod=~"workload-2*"}) and threshold '10'
4. scale workload-1 up to 25 replicas

Logs from KEDA operator

example

KEDA Version

2.3.0

Kubernetes Version

1.19

Platform

Amazon Web Services

Scaler Details

Prometheus

Anything else?

No response

[diegosanchez]

Do you have any news? We got similar behaviour with keda 1.5 as well.

[BorisDr]

@diegosanchez Nope, still waiting

[tomkerkhove]

@zroubalik Is this just because of how we report metrics to HPA or not?

[mishamo]

The prometheus scaler is an "average value" scaler. This means that the value you get from prometheus will be divided by the number of ready pods in the target deployment/job before being passed on to the HPA.

E.g. Your target deployment has 2 pods, prometheus returns 15 for your metric, your threshold is 10. The HPA will show 7500m/10.

Does that explain this issue, or do you think there's still a bug here?

[diegosanchez]

After several trials and errors, we (@ronalcelaya and I) bumped into a helpful piece of documentation: https://github.com/kedacore/keda/blob/v1.5.0/CREATE-NEW-SCALER.md#getmetricspecforscaling.

Could we refer to it from the documentation page?

[tomkerkhove]

Definately, feel free to open a PR!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.