Setting autoscaling.keda.sh/paused: "false" leads to ScaledObject/Job being paused

[nappelson]

Report

I noticed that setting

autoscaling.keda.sh/paused: "false"

pauses the ScaledObject.

Expected Behavior

I would expect setting autoscaling.keda.sh/paused: "false" to turn off the forced pausing behavior.

Actual Behavior

Instead, regardless of what autoscaling.keda.sh/paused is set to, the ScaledObject remains in the paused state. To resume the scaling behavior, I removed the annotation.

Steps to Reproduce the Problem

1. add autoscaling.keda.sh/paused: "false" annotation to an existing, running ScaledObject/ScaledJob
2. run kubectl get ScaledObject and verify that the object is in the paused: True state

Logs from KEDA operator

No response

KEDA Version

2.12.0

Kubernetes Version

1.27

Platform

Google Cloud

Scaler Details

Redis

Anything else?

I just noticed based on the check here -

keda/controllers/keda/scaledobject_controller.go

Line 213 in 25dc205

pausedAnnotationFound := scaledObject.HasPausedAnnotation()

that this is expected behavior. Also, in the documentation this looks like expected behavior as the suggested action to resume the scaling is to remove the annotation - https://keda.sh/docs/2.12/concepts/scaling-deployments/#pause-autoscaling

That being said, the use of true in the documentation is confusing as it implies that switching to false should disable the pausing functionality.

I was going to submit a PR to check the value of the annotation but I'm curious what people think the desired behavior should be.

[zroubalik]

@nappelson you are right, false should definitely unpause the SO/SJ. A PR with a fix (and test) would be much appreciated 🙏

[nappelson]

@nappelson you are right, false should definitely unpause the SO/SJ. A PR with a fix (and test) would be much appreciated 🙏

@zroubalik FWIW, I put up this PR yesterday kedacore/keda-docs#1266. If this becomes the documentation, then enforcing a boolean would become a backwards incompatible change rather than patching a bug. Given how the original documentation is worded (checking the presence of the annotation) technically enforcing a boolean would become backwards incompatible as well (even though no one likely no one is relying on the behavior of setting the annotation to false).

I'm curious, do you think we should enforce a valid boolean value for this annotation and show some warning (while pausing the scaler) if it isn't valid?

[zroubalik]

I would say, that:

• if the annontation is set (without a value) or set to true then pause
• if not set or set to false then unpause

This way we don't break backwards compatibility.

@kedacore/keda-maintainers wdyt?

[JorTurFer]

I would say, that:

• if the annontation is set (without a value) or set to true then pause
• if not set or set to false then unpause

This way we don't break backwards compatibility.

@kedacore/keda-maintainers wdyt?

It's okey to me

[tomkerkhove]

Sounds good to me. @nappelson Are you willing to contribute this?

[tomkerkhove]

if the annontation is set (without a value) or set to true then pause

Even if it's not true it is considered to be pausing it so we should make sure we don't break people (typo or whatever value) or we can block non supported values through webhooks (optional)

[nappelson]

Sounds good to me. @nappelson Are you willing to contribute this?

@tomkerkhove yep I'm happy to put up a PR. May not be able to do it right away (should be by next week though).

@zroubalik

I would say, that:

if the annontation is set (without a value) or set to true then pause
if not set or set to false then unpause
This way we don't break backwards compatibility.

Yep this makes sense to me!

[nappelson]

Note - I discovered this bug while testing code for this issue https://github.com/kedacore/keda/issues?q=is%3Aissue+is%3Aopen+paused

[jgtvares]

I'm also experiencing this!
In my team we mantain a PaaS with 989 apps running today and a lot of them rely on KEDA for CPU and memory scaling.
Is there an ETA for this fix?

[jgtvares]

I'm working around this by setting the annotation as null when I don't want the ScaledObject to be paused.
In my case we have an API that interacts with Kubernetes using an official client library, so setting the value as null results in the annotation being removed when patching the ScaledObject. I didn't test the annotation with the actual null value.