-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prometheus: Scaling to zero not working in KEDA 1.4.0 #770
Comments
Could you please rerun 1.4.0 with debug log level on KEDA operator? https://github.com/kedacore/keda#setting-log-levels (or set it in the chart). |
And paste here your |
I don't see anything suspicious in keda-logs.txt, except the last line KEDA scales replica to 1 from 0 even though the Prometheus metric is zero: |
It seems like this regression was brought by this change: https://github.com/kedacore/keda/pull/695/files#diff-a63ae5a2f6036b9f3bc750d5fe46437cR105 @droessmj what did you do this particular change? ie. Set scaler to active even for value 0? |
@zroubalik That commit updated a zero result to be non-error inducing. Based on the behavior described above I'm assuming since it's now not throwing errors, the scaler ensures 1 replica is up regardless. The "fix" I introduced just returns zero as a valid metric. If we need a special case where zero is non-error inducing but not a valid metric that can be introduced, but as-is the referenced commit just allows this code to now run for zero results:
|
@droessmj I see, but the scaler marks itself as Active even when there is zero result. So my only concern is the change in isActive() function: |
I believe you're correct. Reverting the isActive check while retaining the other part should resolve. |
@hmoravec are you able to retest the change please if I send you link to dev image later today? |
Would be great if anybody with Prometheus instance could check that the fix helped, just replace the images for KEDA Operator and KEDA Metrics Server. Thanks! |
@zroubalik Sure, I'll test it. Btw automatic tests are planned? :-) |
We are always open for PRs ;) But you are right, this shouldn't have slipped through. Sorry about this. |
@zroubalik Working, thanks! It scaled down to zero when the metric became zero and scaled up when the metric became positive. |
@hmoravec great thanks. Yeah we do have some tests but it doesn't cover everything, so this is an area we definitely need to improve. |
Scaling to and from zero is not working with Prometheus scaler after KEDA upgrade from 1.3.0 to 1.4.0.
Expected Behavior
KEDA should scale up deployment from zero when Prometheus metric increases from zero and scale down to zero when the metric decreases to zero.
Actual Behavior
KEDA scales deployment from zero to one replica even when Prometheus metric is zero and never scales it down to zero.
Steps to Reproduce the Problem
minReplicaCount=0
and use Prometheus scaler with query that returns constant zero.and watch that KEDA scales the deployment to zero and again up from zero when the metric becomes positive as expected.
Specifications
The text was updated successfully, but these errors were encountered: