You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
KEDA AWS SQS Scaler fails to account for any in-flight (aka: NotVisible) messages when determining if the ScaledObject should scale-in, and prematurely scales the object down, even if the message is still processing/in-flight.
Expected Behavior
KEDA should wait to scale down until both ApproximateNumberOfMessagesNotVisible and ApproximateNumberOfMessagesNotVisible reach 0, allowing any in-flight messages to complete processing.
Actual Behavior
When there are N messages on the SQS Queue and Keda scales up accordingly, once the last message starts processing the ApproximateNumberOfMessages metric goes to 0, KEDA waits for cooldown period, then scales object back down to 0, interrupting the in-flight processing of the last message.
Note: We renew SQS message leases during long processing, so messages can stay in-flight (aka: NotVisible) for long periods of time, depending on how long it takes to process the message.
We were able to work around this problem temporarily by increasing the cooldown period to ensure that we have enough time to process our messages, however, this is not a good longterm solution as it means frequently leaving pods running much longer than necessary, and if the cooldown is not long enough, can cause us to fail to process the last message on the queue.
Send a message to the Queue which will be processed by the app, but requires more than 5mins to process. (You could just put a sleep in your app, as long as you continue to renew the SQS message lease, the message will remain in-flight). ApproximateNumberOfMessages == 1, ApproximateNumberOfMessagesNotVisible == 0
KEDA will scale up the pod, app will start processing the message, ApproximateNumberOfMessages == 0, ApproximateNumberOfMessagesNotVisible == 1
KEDA will poll next interval and see that ApproximateNumberOfMessages == 0 and begin cooldown period.
At end of cooldown period, KEDA will scale deploy down to minimum, which is 0 in this case, but app was not finished processing. SQS will timeout message (since app is not renewing lease anymore), and message will become visible again.
KEDA will scale up again. This will happen over and over until SQS Message reaches max attempts, and will go into dead letter queue.
Logs from KEDA operator
No response
KEDA Version
2.1.0
Kubernetes Version
1.18
Platform
Amazon Web Services
Scaler Details
AWS SQS Queue
The text was updated successfully, but these errors were encountered:
Report
KEDA AWS SQS Scaler fails to account for any in-flight (aka:
NotVisible
) messages when determining if the ScaledObject should scale-in, and prematurely scales the object down, even if the message is still processing/in-flight.Expected Behavior
KEDA should wait to scale down until both
ApproximateNumberOfMessagesNotVisible
andApproximateNumberOfMessagesNotVisible
reach 0, allowing any in-flight messages to complete processing.Actual Behavior
When there are N messages on the SQS Queue and Keda scales up accordingly, once the last message starts processing the
ApproximateNumberOfMessages
metric goes to 0, KEDA waits for cooldown period, then scales object back down to 0, interrupting the in-flight processing of the last message.Note: We renew SQS message leases during long processing, so messages can stay in-flight (aka: NotVisible) for long periods of time, depending on how long it takes to process the message.
We were able to work around this problem temporarily by increasing the cooldown period to ensure that we have enough time to process our messages, however, this is not a good longterm solution as it means frequently leaving pods running much longer than necessary, and if the cooldown is not long enough, can cause us to fail to process the last message on the queue.
Steps to Reproduce the Problem
ApproximateNumberOfMessages
== 1,ApproximateNumberOfMessagesNotVisible
== 0ApproximateNumberOfMessages
== 0,ApproximateNumberOfMessagesNotVisible
== 1ApproximateNumberOfMessages
== 0 and begin cooldown period.Logs from KEDA operator
No response
KEDA Version
2.1.0
Kubernetes Version
1.18
Platform
Amazon Web Services
Scaler Details
AWS SQS Queue
The text was updated successfully, but these errors were encountered: