Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(logs): log retention custom resource timed out during deploy #26995

Merged
merged 3 commits into from
Sep 6, 2023

Conversation

mrgrain
Copy link
Contributor

@mrgrain mrgrain commented Sep 4, 2023

We use a custom resource to set the log retention for log groups created by the Lambda service.
This custom resource handler code has a built-in retry mechanism to avoid throttling when executing many LogRetention CRs.
Users can customize the number of possible retries, potentially retrying for a long time.
This can cause the situation that further retries should be attempted, but the Lambda Function timeout is exceeded.

The change sets the lambda execution timeout to its maximum value to allow for up to 15 minutes of retries.
If the retry budget is exhausted, the handler will throw an error and exit early.

Closes #24485


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@aws-cdk-automation aws-cdk-automation requested a review from a team September 4, 2023 10:00
@github-actions github-actions bot added bug This issue is a bug. p2 labels Sep 4, 2023
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Sep 4, 2023
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@mrgrain mrgrain changed the title let users set the lambda timeout fix(logs): log retention rate custom resource timed out during deploy Sep 4, 2023
@mrgrain mrgrain changed the title fix(logs): log retention rate custom resource timed out during deploy fix(logs): log retention custom resource timed out during deploy Sep 4, 2023
@mrgrain mrgrain changed the title fix(logs): log retention custom resource timed out during deploy fix(logs): log retention custom resource times out during deploy Sep 4, 2023
@mrgrain mrgrain changed the title fix(logs): log retention custom resource times out during deploy fix(logs): log retention custom resource timed out during deploy Sep 4, 2023
@aws-cdk-automation aws-cdk-automation dismissed their stale review September 4, 2023 10:21

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Sep 4, 2023
@mrgrain mrgrain force-pushed the mrgrain/fix/log-retention-lambda-timeout branch 3 times, most recently from 75ba440 to 19d9dde Compare September 5, 2023 12:39
@mrgrain mrgrain requested a review from rix0rrr September 5, 2023 14:40
@mrgrain mrgrain force-pushed the mrgrain/fix/log-retention-lambda-timeout branch from 0c3f992 to da43a57 Compare September 5, 2023 14:40
@mergify
Copy link
Contributor

mergify bot commented Sep 6, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Sep 6, 2023
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 9cf8df4
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit a73f256 into main Sep 6, 2023
12 checks passed
@mergify mergify bot deleted the mrgrain/fix/log-retention-lambda-timeout branch September 6, 2023 02:13
@mergify
Copy link
Contributor

mergify bot commented Sep 6, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mikewrighton pushed a commit that referenced this pull request Sep 14, 2023
)

We use a custom resource to set the log retention for log groups created by the Lambda service.
This custom resource handler code has a built-in retry mechanism to avoid throttling when executing many LogRetention CRs.
Users can customize the number of possible retries, potentially retrying for a long time.
This can cause the situation that further retries should be attempted, but the Lambda Function timeout is exceeded.

The change sets the lambda execution timeout to its maximum value to allow for up to 15 minutes of retries.
If the retry budget is exhausted, the handler will throw an error and exit early.

Closes #24485

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. contribution/core This is a PR that came from AWS. p2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

custom_resources: log retention rate limit error during deploy
4 participants