Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS Lambda 'recursive loop protection' might consider the lambda function a problem #30

Open
mig5 opened this issue Jul 12, 2023 · 2 comments

Comments

@mig5
Copy link

mig5 commented Jul 12, 2023

I have a variant of this code that has been working well for several years.

In the last 24h I got an email from AWS saying that AWS Lambda is introducing 'recursive loop detection'.

Starting July 5, 2023, AWS Lambda is launching recursive loop detection, a capability that automatically detects and stops certain recursive invocations. With this launch, Lambda will stop invocations for functions that utilize supported SDK versions [1] and are part of a recursive loop composed of the underlying Lambda function and Amazon SQS queue and/or Amazon SNS topic after 16 recursive calls.

To prevent potential disruption to your account, we have turned off this new feature for the accounts in the regions listed in this notification because we previously detected that you have one or more Lambda functions in the xxxxx Region that are being invoked in a recursive loop with other AWS resources. These recursive loops were detected some time during the last three months. Recursive invocations within these accounts will not be stopped.

[1] https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html

Code such as this draining example will be considered by AWS as a recursive loop because it publishes a message to the same SNS topic that launches the lambda function as per here.

You might want to update the example code with a suitable mitigation (perhaps some sort of decreasing counter?)

@mig5
Copy link
Author

mig5 commented Jul 12, 2023

Also, with an autoscale cluster lifecycle hook's default result set to ABANDON with a 900s max heartbeat timeout, I would think that there is already an upper bound on the fact that eventually the instances in the autoscale cluster will get terminated no matter what.

Which means that eventually the lambda function would stop recursing anyway because those instances no longer can be interrogated to see how many tasks are running on them - no?

@mugurelu
Copy link

mugurelu commented Aug 3, 2023

We have the same issue. Would be cool to see an update of this lambda function from this project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants