Editing ScaledObject on Workload in Production

[corradomatt]

Hey all! First, thanks for a great project while I'm not using it in production yet testing is going very well and I'm excited to see it in action.

I've been running load tests using Keda ScaledObjects to control the HPA for my Deployment. I'm using the gcp-stackdriver trigger on a custom GCP log metric from the workload itself and it's been working great. However, I wanted to test the process for making adjustments to the metadata.targetValue of my trigger and noticed something concerning. I have the minReplicaCount currently set to 10 and during a load test when I adjusted the targetValue I noticed that my workload was scaled down from 170 pods to 10 before the new calculation kicked in a scaled it up to the expected 200+ pods. The process happened quickly enough and my load test hardly noticed the issue, but I know this wouldn't be the case in production.

How can I ensure that the HPA doesn't scale down my workload to the minReplicaCount value when making minor adjustments to the ScaleObject?

[corradomatt]

I think that I see the issue in the keda-operator logs...

{
  "error": "rpc error: code = Canceled desc = grpc: the client connection is closing",
  "level": "error", "logger":"gcp_stackdriver_scaler",
  "msg":"error getting metric value",
  ...
}

{
  "error": "scaler with id 0 not found, len = 0, cache has been probably already invalidated",
  "level": "error",
  "logger": "scale_handler",
  "msg": "error getting scale decision",
  ...
}

It looks like the connection to GCP metrics breaks and then the scaler uses the fallback.replicas number. Here's what I've set for the fallback values...

fallback:
    failureThreshold:      3
    replicas:               10

Should I increase the failureThreshold value maybe? Or do I just need to adjust the fallback.replicas to match what is currently running when I make any adjustment?

[corradomatt]

I tried increasing the failureThreshold to 10 but that didn't help unfortunately. For now, if I make changes, I update the fallback.replicas value to whatever the current pod count is for the workload. That helps but I'll keep playing around with the failureThreshold value to see if I can find something that works.

[rickbrouwer]

I will also look later to see if I can find something. I was able to reproduce this.

[rickbrouwer]

Do you use ArgoCD by chance?

[corradomatt]

I do! BUT in this case ArgoCD wasn't controlling the ScaledObject yet. I was managing the ScaledObject by hand during testing and all edits were being done with kubectl edit scaledobject X.

[rickbrouwer]

Thanks for the explanation. I've had a look at the code and done some tests.
I think it's possible to modify the doFallbackScaling function in the scale_scaledobjects.go. That would mean a new parameter that indicates whether you want to use the currentReplicas of that moment so that it doesn't scale back but keeps the current number. It will look something like this:

fallback:
    failureThreshold: 3
    replicas: 10
    useCurrentReplicas: true

Or maybe something like useCurrentAsMinimum, so that it always uses the current number unless the replicas from the fallback are higher.

Would this be something that would help you @corradomatt ?
I don't know if this is a desirable option, otherwise I might be able to make a contribution. WDYT @zroubalik ?