feat(custom-resources): add logging property to `AwsSdkCall` and create `Logging` class #29648

colifran · 2024-03-29T08:30:41Z

Reason for this change

SDK v2 and v3 handlers for AwsCustomResource log the event object passed to the handler, API responses, and caught /uncaught errors for each SDK call made. This can potentially result in logging sensitive information that a user may wish to hide. This PR introduces a new logging property on the AwsSdkCall interface that can be used to provide more control over logging in the SDK v2 and v3 handlers on a per SDK call basis. The logging flag is configurable via a new Logging class which exposes two static methods:

all: all logging during lambda execution is turned on
withDataHidden: hides all logged data associated with the API call response. This includes the raw response as well as the Data field on the response object

Additional logging configurations can be added in the future.

Description of changes

Added a logging flag to the AwsSdkCall interface which is configurable via the new Logging class. The Logging class has an internal render method which renders the specified logging configuration which is passed as part of the create, update, and delete ResourceProperties to the lambda handler. These logging properties are then used throughout the handler to control what is logged based on their value

Description of how you validated changes

A new integ test with logging as withDataHidden was added
Unit tests to ensure calling render on a Logging instance produces the expected result
Unit tests to ensure that using logging with AwsSdkCall while using AwsCustomResource produces the correct CloudFormation template

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

vinayak-kukreja

I agree with the changes. But, we should get consensus among the team too about this. Especially stakeholders you have already synced with about this.

vinayak-kukreja · 2024-04-01T23:07:59Z

packages/aws-cdk-lib/custom-resources/lib/aws-custom-resource/aws-custom-resource.ts

+  /**
+   * Whether or not to include API response data in the logs for the underlying lambda.
+   *
+   * If this value is false, the raw API response and the `Data` field in the response


I think we should add an example of Response Object here so that we explicitly mention what we will still be logging with this. And also add this to our documentation

vinayak-kukreja · 2024-04-01T23:08:37Z

packages/aws-cdk-lib/custom-resources/README.md

-### Custom Resource Examples
+### Custom Resource Logging
+
+The underlying lamdda function used by the `AwsCustomResource` construct logs the following information:


NIT

Suggested change

The underlying lamdda function used by the `AwsCustomResource` construct logs the following information:

The underlying lambda function used by the `AwsCustomResource` construct logs the following information:

vinayak-kukreja · 2024-04-01T23:09:20Z

packages/aws-cdk-lib/custom-resources/README.md

@@ -607,9 +607,16 @@ new cr.AwsCustomResource(this, 'ListObjects', {
 Note that even if you restrict the output of your custom resource you can still use any
 path in `PhysicalResourceId.fromResponse()`.

-### Custom Resource Examples
+### Custom Resource Logging


Lets add an example Response Object in these docs

Good call out. This has been updated.

TheRealAmazonKendra · 2024-04-03T01:43:18Z

Did you close the original PR altogether? I was looking for my comments on it but couldn't find them. This still has the issue of this being a boolean without an option for additional customization.

colifran · 2024-04-03T04:42:16Z

@TheRealAmazonKendra Can you provide more detail on why you think that? We can provide this flag and still keep the door open for additional customization. This flag would only control the logging of the Data field in the response object. Any additional customization could come with a future PR and this flag could be set or unset without blocking a future property that focuses on customization.

Say, for example, we wanted to add something to control log levels. If logApiResponseData is set to false and the log level is DEBUG then CloudWatch Logs still has all errors (caught and uncaught), the event received by the handler, and the response object (just without the Data field). That would open the door for a user to have all the benefits of a DEBUG log level while making an API call that returns sensitive data. Except now they don't have to worry about exposing data in CloudWatch Logs that they want hidden. If that isn't an issue for them then they can ignore logApiResponseData or explicitly set it to true.

The previous PR is linked here. The flag proposed in that PR was too overbearing and would have blocked us from implementing additional customization in the future (as you correctly stated). I don't think this has that problem.

Signed-off-by: Francis <colifran@amazon.com>

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

colifran · 2024-04-04T02:38:24Z

@TheRealAmazonKendra pushed initial changes we discussed offline. To summarize we should provide a Logging class that consolidates all of logging levels into an enum-like class. Additionally, this doesn't need to follow standard logging convention (info, trace, debug, etc.). We can provide something like on, off, and selective where selective would give granular control over logged data to the user.

Signed-off-by: Francis <colifran@amazon.com>

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

Signed-off-by: Francis <colifran@amazon.com>

mergify · 2024-04-14T00:24:11Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

…sSdkCall` in unit tests (#29860) ### Reason for this change #29648 introduced a change to the `AwsSdkCall` representation used in the v2 and v3 handler code. Our handler unit tests use `satisfies` to validate that the event object satisfies `AwsSdkCall`. All unit tests and the build still pass, but the linter calls out that the event object doesn't actually satisfy `AwsSdkCall`. #29845 removed the dependency `@aws-cdk/custom-resource-handlers` had on `aws-sdk`. We should add this as devDependency since we're using `aws-sdk` in v2 handler mocks. ### Description of changes I added `logApiResponseData` property to the event objects being tested to make the event satisfy `AwsSdkCall`. I added `aws-sdk` as a dev dependency. We will remove this as part of the v2 handler removal. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

…ce event properties by default (#30418) Closes #30121, #29949 ### Reason for this change PR #29648 introduced a new resource property `logApiResponseData`. This resource property is `true` by default which forces an update for `AwsCustomResource`. For users without `onUpdate` configured an empty data object is returned if no SDK call is configured. This can cause an attribute error if the user is depending on a data from a specific SDK call. ### Description of changes Made `logApiResponseData` undefined by default which will not trigger `onUpdate`. To maintain backwards compatibility with the original PR introducing `logApiResponseData` as true by default, I've also introduced a feature flag that will allow users to keep the current behavior so they aren't now forced into another `onUpdate` event. ### Description of how you validated changes Updated unit tests where `logApiResponseData` was added as a resource property. Added new unit test to verify that `logApiResponseData` could be added to the event. Updated unit tests that test `_render()` to ensure that the default case will result in an empty object. Updated integ tests. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

…ce event properties by default (aws#30418) Closes aws#30121, aws#29949 ### Reason for this change PR aws#29648 introduced a new resource property `logApiResponseData`. This resource property is `true` by default which forces an update for `AwsCustomResource`. For users without `onUpdate` configured an empty data object is returned if no SDK call is configured. This can cause an attribute error if the user is depending on a data from a specific SDK call. ### Description of changes Made `logApiResponseData` undefined by default which will not trigger `onUpdate`. To maintain backwards compatibility with the original PR introducing `logApiResponseData` as true by default, I've also introduced a feature flag that will allow users to keep the current behavior so they aren't now forced into another `onUpdate` event. ### Description of how you validated changes Updated unit tests where `logApiResponseData` was added as a resource property. Added new unit test to verify that `logApiResponseData` could be added to the event. Updated unit tests that test `_render()` to ensure that the default case will result in an empty object. Updated integ tests. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

aws-cdk-automation · 2024-07-25T21:19:29Z

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

github-actions bot added the p2 label Mar 29, 2024

aws-cdk-automation requested a review from a team March 29, 2024 08:30

mergify bot added the contribution/core This is a PR that came from AWS. label Mar 29, 2024

This was referenced Apr 1, 2024

Monthly PR metrics report #29668

Closed

Weekly PR metrics report #29673

Closed

colifran changed the title ~~feat(custom-resources): add property to AwsCustomResourceProps to prevent logging API call response data~~ feat(custom-resources): add optional property to AwsCustomResourceProps to prevent logging API call response data Apr 1, 2024

vinayak-kukreja previously requested changes Apr 1, 2024

View reviewed changes

logging

33c7ac8

Signed-off-by: Francis <colifran@amazon.com>

colifran force-pushed the colifran/response-data branch from 33b18df to 33c7ac8 Compare April 4, 2024 02:35

aws-cdk-automation previously requested changes Apr 4, 2024

View reviewed changes

colifran added 2 commits April 3, 2024 19:46

v2 handler

3e0e2e1

Signed-off-by: Francis <colifran@amazon.com>

docstring

e87c3e6

Signed-off-by: Francis <colifran@amazon.com>

colifran changed the title ~~feat(custom-resources): add optional property to AwsCustomResourceProps to prevent logging API call response data~~ feat(custom-resources): add logging property to AwsCustomResourceProps and create Logging class Apr 4, 2024

colifran added 12 commits April 3, 2024 20:33

make logging abstract and implement render

415b0c6

Signed-off-by: Francis <colifran@amazon.com>

unit tests

dcc0dc8

Signed-off-by: Francis <colifran@amazon.com>

unit test for off

4e61b2d

Signed-off-by: Francis <colifran@amazon.com>

log sdk version

c8c0b64

Signed-off-by: Francis <colifran@amazon.com>

README

bcc1ab8

Signed-off-by: Francis <colifran@amazon.com>

readme

97660cd

Signed-off-by: Francis <colifran@amazon.com>

logging integ tests

d20b6b4

Signed-off-by: Francis <colifran@amazon.com>

integ test

ea8d4de

Signed-off-by: Francis <colifran@amazon.com>

selective logging should select things to log

840746a

Signed-off-by: Francis <colifran@amazon.com>

align readme

63914b9

Signed-off-by: Francis <colifran@amazon.com>

logging options

da86462

Signed-off-by: Francis <colifran@amazon.com>

snaps

43bdef9

Signed-off-by: Francis <colifran@amazon.com>

class doc string

af67a3b

Signed-off-by: Francis <colifran@amazon.com>

colifran removed the pr/do-not-merge This PR should not be merged at this time. label Apr 14, 2024

mergify bot merged commit b049064 into main Apr 14, 2024
15 checks passed

mergify bot deleted the colifran/response-data branch April 14, 2024 00:24

colifran mentioned this pull request Apr 17, 2024

chore: add aws-sdk as dev dependency for unit tests and satisfy AwsSdkCall in unit tests #29860

Merged

1 task

p3tek mentioned this pull request May 9, 2024

custom-resources: PR:29648 caused Custom resources to be replaced #30121

Closed

This was referenced May 23, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#98

Open

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#99

Open

rwlxxvii mentioned this pull request May 27, 2024

[Snyk] Upgrade aws-cdk-lib from 2.136.1 to 2.140.0 rwlxxvii/containers#124

Merged

NOUIY mentioned this pull request May 30, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#101

Open

colifran mentioned this pull request Jun 1, 2024

fix(custom-resources): logApiResponseData is added to custom resource event properties by default #30418

Merged

1 task

NOUIY mentioned this pull request Jun 7, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#102

Open

niravparikh05 mentioned this pull request Jun 13, 2024

[Snyk] Upgrade aws-cdk from 2.65.0 to 2.142.1 paralus/eks-blueprints-addon#3

Open

NOUIY mentioned this pull request Jun 14, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#103

Open

snyk-io bot mentioned this pull request Jun 20, 2024

[Snyk] Upgrade aws-cdk-lib from 2.138.0 to 2.143.1 Opetushallitus/heratepalvelu#280

Open

NOUIY mentioned this pull request Jun 20, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#104

Open

NOUIY mentioned this pull request Jun 22, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#105

Open

kazimanzurrashid mentioned this pull request Jun 29, 2024

[Snyk] Upgrade aws-cdk-lib from 2.144.0 to 2.145.0 kazimanzurrashid/aws-scheduler-go#821

Open

This was referenced Jun 29, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#106

Open

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#107

Open

This was referenced Jul 12, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#108

Open

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#109

Open

This was referenced Jul 19, 2024

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#110

Open

[Snyk] Upgrade: , aws-cdk NOUIY/aws-solutions-constructs#111

Open

aws locked as resolved and limited conversation to collaborators Jul 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(custom-resources): add logging property to `AwsSdkCall` and create `Logging` class #29648

feat(custom-resources): add logging property to `AwsSdkCall` and create `Logging` class #29648

colifran commented Mar 29, 2024 •

edited

Loading

vinayak-kukreja left a comment

vinayak-kukreja Apr 1, 2024

colifran Apr 1, 2024

vinayak-kukreja Apr 1, 2024

colifran Apr 1, 2024

vinayak-kukreja Apr 1, 2024

colifran Apr 1, 2024

TheRealAmazonKendra commented Apr 3, 2024

colifran commented Apr 3, 2024

aws-cdk-automation left a comment

colifran commented Apr 4, 2024

mergify bot commented Apr 14, 2024

aws-cdk-automation commented Jul 25, 2024

	The underlying lamdda function used by the `AwsCustomResource` construct logs the following information:
	The underlying lambda function used by the `AwsCustomResource` construct logs the following information:

feat(custom-resources): add logging property to AwsSdkCall and create Logging class #29648

feat(custom-resources): add logging property to AwsSdkCall and create Logging class #29648

Conversation

colifran commented Mar 29, 2024 • edited Loading

Reason for this change

Description of changes

Description of how you validated changes

Checklist

vinayak-kukreja left a comment

Choose a reason for hiding this comment

vinayak-kukreja Apr 1, 2024

Choose a reason for hiding this comment

colifran Apr 1, 2024

Choose a reason for hiding this comment

vinayak-kukreja Apr 1, 2024

Choose a reason for hiding this comment

colifran Apr 1, 2024

Choose a reason for hiding this comment

vinayak-kukreja Apr 1, 2024

Choose a reason for hiding this comment

colifran Apr 1, 2024

Choose a reason for hiding this comment

TheRealAmazonKendra commented Apr 3, 2024

colifran commented Apr 3, 2024

aws-cdk-automation left a comment

Choose a reason for hiding this comment

colifran commented Apr 4, 2024

mergify bot commented Apr 14, 2024

aws-cdk-automation commented Jul 25, 2024

feat(custom-resources): add logging property to `AwsSdkCall` and create `Logging` class #29648

feat(custom-resources): add logging property to `AwsSdkCall` and create `Logging` class #29648

colifran commented Mar 29, 2024 •

edited

Loading