fix(core): fix rate limit errors when deploying cloudwatch log groups

[xuetinp]

Problem #807

When the RFDK creates a CloudWatch Log group it does it via the CDK's LogRetention construct. The CloudWatch Logs APIs that the LogRetention construct calls have a very low TPS limit (5 TPS). The result is that when deploying multiple RFDK constructs one can encounter rate limit errors in the deployment -- this is particularly seen, randomly, with the RFDK's integration tests.

Solution

CDK's LogRetention construct provides a way - logRetentionRetryOptions to configure the Javascript SDK that it's using to make the CloudWatch API calls. Using this, the RFDK should set higher max retries and retry delay. Currently default backoff retry base is 100ms and max retry count is 3, proposed value is 200ms and 7. This is a good combination as it achieves a lot of retries while the total function runtime is not too long.

• Added retry option to all places where new LogRetention is created.
• Extended log retention function timeout. Referencing https://github.com/aws/aws-cdk/blob/205e493e7bd6c5212f0ae374fdee28128ea49afe/packages/%40aws-cdk/aws-logs/lib/log-retention.ts#L122-L130

Testing

• Created a CloudFormation app that iteratively initialize LogRetention constructs. Wrote a script to deploy constructs in parallel to reproduce the rate limit error. Verified proposed values for max retry count and retry delay prevent the error.
• Added unit test to verify template is modified as expected.
• Build success.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[ddneilson]

This is perfect. Thank you!

[jusiskin]

Great work. I have two minor suggestions to improve this.

[jericht]

LGTM!