fix: stop scale loop for pause Scaledbject (issue #4253) #4550

tobotg · 2023-05-18T12:49:56Z

In the controller in the reconcile loop, when a scaledObject is paused (annotated with ), this fix stops the scaling logic (to prevent attempt to connect the to scalers), and deletes the generated HPA (so it is not querying KEDA Metric Server for metrics)

Checklist

I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #4253

tobotg · 2023-05-18T12:52:57Z

@zroubalik @tomkerkhove Resubmitting a new PR.

zroubalik

Great, could you please update this PR to contain newest commits? there are some improvements in e2e tests logging, thanks!

zroubalik · 2023-05-18T13:30:19Z

/run-e2e
Update: You can check the progress here

zroubalik

Thanks for the contribution, there are some minor nits and we should also introduce a new Condition in the ScaledObject.Status.

controllers/keda/scaledobject_controller.go

apis/keda/v1alpha1/condition_types.go

zroubalik

Looking good, could you please add a marker to CRD to show PAUSED column and regenerate the CRD (make generate)?
Probably before the AGE column:
https://github.com/kedacore/keda/blob/main/apis/keda/v1alpha1/scaledobject_types.go#L36

zroubalik · 2023-05-31T08:25:43Z

/run-e2e
Update: You can check the progress here

tobotg · 2023-05-31T17:33:07Z

Looking good, could you please add a marker to CRD to show PAUSED column and regenerate the CRD (make generate)?
Probably before the AGE column:
https://github.com/kedacore/keda/blob/main/apis/keda/v1alpha1/scaledobject_types.go#L36

Sure thing. I added it and run make manifests.

zroubalik · 2023-06-01T16:04:13Z

/run-e2e
Update: You can check the progress here

zroubalik

the e2e test for paused feature fails :(

tobotg · 2023-06-01T21:20:23Z

the e2e test for paused feature fails :(

ok thanks @zroubalik , I'll have a look at the pause_scaling_test logs, I guess this is what failed (I can't seem to find the details of the failure in the GitHub actions link)

tobotg · 2023-06-04T19:01:28Z

@zroubalik The e2e tests are passing in my cluster, but I suspect that the test needs more time to recover the scaling loop (loads might be inducing some more delay), so I'm increasing the wait time. Could you please trigger another run when you get a chance? Thanks

zroubalik · 2023-06-05T09:25:04Z

/run-e2e
Update: You can check the progress here

zroubalik · 2023-06-05T13:20:29Z

/run-e2e pause
Update: You can check the progress here

tobotg · 2023-06-05T15:32:37Z

@zroubalik

/run-e2e pause
Update: You can check the progress here

That's weird. I'll need to have a deeper look later, as I'm at loss understanding the failures here.

zroubalik · 2023-06-05T17:10:46Z

@zroubalik

/run-e2e pause
Update: You can check the progress here

That's weird. I'll need to have a deeper look later, as I'm at loss understanding the failures here.

Seems like the testing deployment is not even initally scaled out, it stays on 0 replicas.

tobotg · 2023-06-06T04:48:13Z

@zroubalik

/run-e2e pause
Update: You can check the progress here

That's weird. I'll need to have a deeper look later, as I'm at loss understanding the failures here.

Seems like the testing deployment is not even initally scaled out, it stays on 0 replicas.

Yes, that's exactly what is puzzling me. Need to sort it out!

tobotg · 2023-06-08T00:43:01Z

I made two changes to the test

use a much smaller nginx image to speed up deployment
make the test wait for the monitored deployment to hit its target replica count before proceeding
@zroubalik

zroubalik · 2023-06-08T07:35:57Z

/run-e2e pause
Update: You can check the progress here

zroubalik · 2023-06-08T09:52:08Z

@tobotg I ran your branch locally and got the same error as we can see here in the PR.

tobotg · 2023-06-08T10:12:38Z

@tobotg I ran your branch locally and got the same error as we can see here in the PR.

@zroubalik Interesting that you're able to reproduce it locally, as I'm not able to reproduce it here. Any chance you can share Keda logs and app:pause-scaling-test and app:pause-scaling-test-monitored logs? That could be helpful for debugging. Thanks in advance.

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

…or msg and new condition msg Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

…sed condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

… get hpa name and paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

…ifest for paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

…rease test wait timeout Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

…ed up test with slim container image and wait for monitored deployment Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

tobotg · 2023-06-14T23:25:52Z

@zroubalik I had some time tonight to look into this issue, and submitted changes to the PR. Could you please trigger an e2e test when you get a chance? Thanks

zroubalik · 2023-06-15T08:14:21Z

/run-e2e pause
Update: You can check the progress here

zroubalik · 2023-06-18T10:25:46Z

/run-e2e
Update: You can check the progress here

zroubalik

/lgtm

Thanks @tobotg! Awesome job, it is a great improvement!

…acore#4550) Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

tobotg requested a review from a team as a code owner May 18, 2023 12:49

zroubalik reviewed May 18, 2023

View reviewed changes

tobotg force-pushed the fix-issue-4253 branch from 1927767 to 2db84cd Compare May 18, 2023 13:07

zroubalik reviewed May 18, 2023

View reviewed changes

tobotg force-pushed the fix-issue-4253 branch from 2db84cd to 2c7efeb Compare May 18, 2023 20:25

zroubalik reviewed May 19, 2023

View reviewed changes

apis/keda/v1alpha1/condition_types.go Show resolved Hide resolved

tobotg force-pushed the fix-issue-4253 branch from 2c7efeb to 4e326f1 Compare May 19, 2023 21:57

tobotg force-pushed the fix-issue-4253 branch from 764adb9 to 1e3c200 Compare May 27, 2023 19:47

zroubalik reviewed May 31, 2023

View reviewed changes

tobotg force-pushed the fix-issue-4253 branch from 1e3c200 to 9493f34 Compare May 31, 2023 17:29

zroubalik reviewed Jun 1, 2023

View reviewed changes

tobotg force-pushed the fix-issue-4253 branch from 9493f34 to fee6e58 Compare June 4, 2023 18:55

zroubalik mentioned this pull request Jun 5, 2023

feat: Provide support for explicitly pausing autoscaling of ScaledJobs #4558

Merged

7 tasks

tobotg force-pushed the fix-issue-4253 branch from fee6e58 to 5faf0b1 Compare June 8, 2023 00:36

tobotg added 8 commits June 15, 2023 01:03

fix: stop scale loop for pause Scaledbject (kedacore#4253)

9afb656

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

fix: stop scale loop for pause Scaledbject (issue kedacore#4253). err…

933e30c

…or msg and new condition msg Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

fix: stop scale loop for pause Scaledbject (issue kedacore#4253). pau…

059dc84

…sed condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

fix: stop scale loop for pause Scaledbject (issue kedacore#4253). fix…

f1a994e

… get hpa name and paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

fix: stop scale loop for pause Scaledbject (issue kedacore#4253). man…

9d37914

…ifest for paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

fix: stop scale loop for pause Scaledbject (issue kedacore#4253). inc…

7039451

…rease test wait timeout Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

fix: stop scale loop for pause Scaledbject (issue kedacore#4253). spe…

fc3dad7

…ed up test with slim container image and wait for monitored deployment Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

fix: stop scale loop for pause Scaledbject (issue kedacore#4253)

937cb53

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

tobotg force-pushed the fix-issue-4253 branch from 5faf0b1 to 937cb53 Compare June 14, 2023 23:05

zroubalik approved these changes Jun 18, 2023

View reviewed changes

zroubalik merged commit 93851e7 into kedacore:main Jun 18, 2023

zroubalik mentioned this pull request Jun 21, 2023

UnPause does not happen immediately #4642

Closed

SpiritZhou pushed a commit to SpiritZhou/keda that referenced this pull request Jun 30, 2023

fix: stop scale loop for pause Scaledbject (issue kedacore#4253) (ked…

bae2640

…acore#4550) Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: stop scale loop for pause Scaledbject (issue #4253) #4550

fix: stop scale loop for pause Scaledbject (issue #4253) #4550

tobotg commented May 18, 2023

tobotg commented May 18, 2023

zroubalik left a comment

zroubalik commented May 18, 2023 •

edited by github-actions bot

Loading

zroubalik left a comment

zroubalik left a comment •

edited

Loading

zroubalik commented May 31, 2023 •

edited by github-actions bot

Loading

tobotg commented May 31, 2023

zroubalik commented Jun 1, 2023 •

edited by github-actions bot

Loading

zroubalik left a comment

tobotg commented Jun 1, 2023

tobotg commented Jun 4, 2023

zroubalik commented Jun 5, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 5, 2023 •

edited by github-actions bot

Loading

tobotg commented Jun 5, 2023

zroubalik commented Jun 5, 2023

tobotg commented Jun 6, 2023

tobotg commented Jun 8, 2023

zroubalik commented Jun 8, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 8, 2023 •

edited

Loading

tobotg commented Jun 8, 2023

tobotg commented Jun 14, 2023

zroubalik commented Jun 15, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 18, 2023 •

edited by github-actions bot

Loading

zroubalik left a comment

fix: stop scale loop for pause Scaledbject (issue #4253) #4550

fix: stop scale loop for pause Scaledbject (issue #4253) #4550

Conversation

tobotg commented May 18, 2023

Checklist

tobotg commented May 18, 2023

zroubalik left a comment

Choose a reason for hiding this comment

zroubalik commented May 18, 2023 • edited by github-actions bot Loading

zroubalik left a comment

Choose a reason for hiding this comment

zroubalik left a comment • edited Loading

Choose a reason for hiding this comment

zroubalik commented May 31, 2023 • edited by github-actions bot Loading

tobotg commented May 31, 2023

zroubalik commented Jun 1, 2023 • edited by github-actions bot Loading

zroubalik left a comment

Choose a reason for hiding this comment

tobotg commented Jun 1, 2023

tobotg commented Jun 4, 2023

zroubalik commented Jun 5, 2023 • edited by github-actions bot Loading

zroubalik commented Jun 5, 2023 • edited by github-actions bot Loading

tobotg commented Jun 5, 2023

zroubalik commented Jun 5, 2023

tobotg commented Jun 6, 2023

tobotg commented Jun 8, 2023

zroubalik commented Jun 8, 2023 • edited by github-actions bot Loading

zroubalik commented Jun 8, 2023 • edited Loading

tobotg commented Jun 8, 2023

tobotg commented Jun 14, 2023

zroubalik commented Jun 15, 2023 • edited by github-actions bot Loading

zroubalik commented Jun 18, 2023 • edited by github-actions bot Loading

zroubalik left a comment

Choose a reason for hiding this comment

zroubalik commented May 18, 2023 •

edited by github-actions bot

Loading

zroubalik left a comment •

edited

Loading

zroubalik commented May 31, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 1, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 5, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 5, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 8, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 8, 2023 •

edited

Loading

zroubalik commented Jun 15, 2023 •

edited by github-actions bot

Loading

zroubalik commented Jun 18, 2023 •

edited by github-actions bot

Loading