feat(custom-resources): provider framework (#4572)

* feat(custom-resources): async custom resources

`AsyncCustomResource` is a framework for defining robust custom resources. It makes it easy to implement providers for resources that require waiting for the resource to be ready, potentially for time period longer than the AWS Lambda timeout (15min at the moment). When a resource is created, the "begin" operation is invoked, and a state machine is triggered to wait for "complete" to indicate that it's done. It can also indicate that it's still waiting, in which case, Step Functions retry logic will kick in and "complete" will be called thereafter. If retries are exhausted, the resource operation will fail with a timeout. Any exceptions thrown from "begin" or "complete" will be reported as an operation failure to ensure that the custom resource does not hang the stack operation.

* misc

- change semantics to "onEvent" and "isComplete"
- use Lambda layers to load user code
- reorg project files

* move aws-custom-resource to a subdirectory (both lib/ and test/)

* good test coverage for async custom resource runtime handler

* misc

* support specifying execution role for each resource instance (very powerful)
* always require that on-event will return a physical resource id (to avoid errors)
* emit entire stack trace on errors
* implement an integration test/sample (S3File)

* add failing test

* remove support for resource-specific roles

* extract user handlers to separate lambdas, which means we are now language agnostic!

* Misc

* use RequestId as the name of the state machine execution due to limitations in the name
* make handler names required to make sure users are aware that they have to implement them
* implement another custom resource in the integration test which "waits" for the object to contain certain contents (tests async resources).

* rename to ProviderFramework and add README

* add types.d

* a bunch of minor fixes

* chore(custom-resources): reorg and migreate to jest

As a preparation for adding the custom resource provider framework (#4572), which includes multiple files and tests, reorg the file structure of this module such that all files
related to the `AwsCustomResource` construct will be under `lib/aws-custom-resource` and `test/aws-custom-resource`.

Also, migrate all unit tests from nodeunit to jest.

* Delete sdk-api-metadata.json

* Update packages/@aws-cdk/custom-resources/lib/provider-framework/provider.ts

Co-Authored-By: Jonathan Goldwasser <jogold@users.noreply.github.com>

* Update packages/@aws-cdk/custom-resources/README.md

Co-Authored-By: Jonathan Goldwasser <jogold@users.noreply.github.com>

* Update packages/@aws-cdk/custom-resources/README.md

Co-Authored-By: Jonathan Goldwasser <jogold@users.noreply.github.com>

* Apply suggestions from code review

Co-Authored-By: Jonathan Goldwasser <jogold@users.noreply.github.com>

* add sdk-json to .gitignore

* user-defined handlers are now passed as lambda.IFunction and make isComplete optional

Instead of presuming to define the AWS Lambda function on behalf of the user, we now simply ask the user to supply a couple of IFunction objects. This gives users maximum flexibility in configuring their AWS Lambda environments.

Furthermore, this change also makes isComplete optional and only defines the waiter resources in case isComplete is provided.

Refactored integ test so that providers are deployed as a separate stack and their entrypoints are exported through CFN export names.

* stop calling isComplete synchronously

Since we now have a better way to represent that onEvent is synchronous (by simply not supplying an isComplete handler), and this will also result in a cleaner stack (no step functions, etc), the value of calling isComplete synchronously has been reduced. Combined with the observation by @jogold that the framework onEvent timeout would have to be user onEvent+isComplete, the complexity is not worth it.

Also, refactor unit test harness such that it simulates the entire workflow (incl. reaction to startExecution and simulation of the waiter step function).

* allow extensibility for CustomResourceProvider

accept an interface instead of a class and use the `bind` pattern to allow defining resources as part of the provider.

* simplify integration test (a single stack)

* update README

* code review fixes

- remove validation of CFN event (not required)
- redesign `failOnError` as `safeHandler` which returns a "safe" cloudformation handler.
- fail if user handler returns a non-JSON response.

* fix docstring in aws-cloudformation

* set all framework timeouts to 15min

later on we can figure out something smarter if need be

* add a section about error handling

* pass modules in topological order during packaging

packages/@aws-cdk/aws-cloudformation/lib/custom-resource.ts

@@ -1,27 +1,76 @@
 import lambda = require('@aws-cdk/aws-lambda');
 import sns = require('@aws-cdk/aws-sns');
-import { CfnResource, Construct, RemovalPolicy, Resource } from '@aws-cdk/core';
+import { CfnResource, Construct, RemovalPolicy, Resource, Token } from '@aws-cdk/core';
 import { CfnCustomResource } from './cloudformation.generated';
 
 /**
  * Collection of arbitrary properties
  */
 export type Properties = {[key: string]: any};
 
-export class CustomResourceProvider {
+/**
+ * Configuration options for custom resource providers.
+ */
+export interface CustomResourceProviderConfig {
+  /**
+   * The ARN of the SNS topic or the AWS Lambda function which implements this
+   * provider.
+   */
+  readonly serviceToken: string;
+}
+
+/**
+ * Represents a provider for an AWS CloudFormation custom resources.
+ */
+export interface ICustomResourceProvider {
+  /**
+   * Called when this provider is used by a `CustomResource`.
+   * @param scope The resource that uses this provider.
+   * @returns provider configuration
+   */
+  bind(scope: Construct): CustomResourceProviderConfig;
+}
+
+/**
+ * Represents a provider for an AWS CloudFormation custom resources.
+ */
+export class CustomResourceProvider implements ICustomResourceProvider {
   /**
    * The Lambda provider that implements this custom resource.
    *
    * We recommend using a lambda.SingletonFunction for this.
    */
-  public static lambda(handler: lambda.IFunction) { return new CustomResourceProvider(handler.functionArn); }
+  public static fromLambda(handler: lambda.IFunction): CustomResourceProvider {
+    return new CustomResourceProvider(handler.functionArn);
+  }
 
   /**
    * The SNS Topic for the provider that implements this custom resource.
    */
-  public static topic(topic: sns.ITopic) { return new CustomResourceProvider(topic.topicArn); }
+  public static fromTopic(topic: sns.ITopic): CustomResourceProvider {
+    return new CustomResourceProvider(topic.topicArn);
+  }
+
+  /**
+   * Use AWS Lambda as a provider.
+   * @deprecated use `fromLambda`
+   */
+  public static lambda(handler: lambda.IFunction) { return this.fromLambda(handler); }
 
-  private constructor(public readonly serviceToken: string) {}
+  /**
+   * Use an SNS topic as the provider.
+   * @deprecated use `fromTopic`
+   */
+  public static topic(topic: sns.ITopic) { return this.fromTopic(topic); }
+
+  /**
+   * @param serviceToken the ServiceToken which contains the ARN for this provider.
+   */
+  private constructor(public readonly serviceToken: string) { }
+
+  public bind(_: Construct): CustomResourceProviderConfig {
+    return { serviceToken: this.serviceToken };
+  }
 }
 
 /**
@@ -31,10 +80,25 @@ export interface CustomResourceProps {
   /**
    * The provider which implements the custom resource
    *
-   * @example CustomResourceProvider.lambda(myFunction)
-   * @example CustomResourceProvider.topic(myTopic)
+   * @example invoke an AWS Lambda function when a lifecycle event occurs
+   *
+   *    CustomResourceProvider.fromLambda(myFunction)
+   *
+   * @example publish lifecycle events to an SNS topic
+   *
+   *    CustomResourceProvider.fromTopic(myTopic)
+   *
+   * @example use the custom resource provider framework
+   *
+   *    import cr = require('@aws-cdk/custom-resources');
+   *    const myProvider = new cr.Provider(...)
+   *
+   *    new cfn.CustomResource(this, 'myResource', {
+   *      provider: myProvider
+   *    });
+   *
    */
-  readonly provider: CustomResourceProvider;
+  readonly provider: ICustomResourceProvider;
 
   /**
    * Properties to pass to the Lambda
@@ -88,16 +152,18 @@ export class CustomResource extends Resource {
     super(scope, id);
 
     const type = renderResourceType(props.resourceType);
-
+    const providerConfig = props.provider.bind(this);
     this.resource = new CfnResource(this, 'Default', {
       type,
       properties: {
-        ServiceToken: props.provider.serviceToken,
+        ServiceToken: providerConfig.serviceToken,
         ...uppercaseProperties(props.properties || {})
       }
     });
 
-    this.resource.applyRemovalPolicy(props.removalPolicy, { default: RemovalPolicy.DESTROY });
+    this.resource.applyRemovalPolicy(props.removalPolicy, {
+      default: RemovalPolicy.DESTROY
+    });
   }
 
   /**
@@ -108,12 +174,29 @@ export class CustomResource extends Resource {
   }
 
   /**
-   * An attribute of this custom resource
-   * @param attributeName the attribute name
+   * Returns the value of an attribute of the custom resource of an arbitrary
+   * type. Attributes are returned from the custom resource provider through the
+   * `Data` map where the key is the attribute name.
+   *
+   * @param attributeName the name of the attribute
+   * @returns a token for `Fn::GetAtt`. Use `Token.asXxx` to encode the returned `Reference` as a specific type or
+   * use the convinience `getAttString` for string attributes.
    */
   public getAtt(attributeName: string) {
     return this.resource.getAtt(attributeName);
   }
+
+  /**
+   * Returns the value of an attribute of the custom resource of type string.
+   * Attributes are returned from the custom resource provider through the
+   * `Data` map where the key is the attribute name.
+   *
+   * @param attributeName the name of the attribute
+   * @returns a token for `Fn::GetAtt` encoded as a string.
+   */
+  public getAttString(attributeName: string): string {
+    return Token.asString(this.getAtt(attributeName));
+  }
 }
 
 /**

packages/@aws-cdk/aws-cloudformation/lib/nested-stack.ts

@@ -5,6 +5,8 @@ import { CfnStack } from './cloudformation.generated';
 const NESTED_STACK_SYMBOL = Symbol.for('@aws-cdk/aws-cloudformation.NestedStack');
 
 /**
+ * Initialization props for the `NestedStack` construct.
+ *
  * @experimental
  */
 export interface NestedStackProps {
@@ -133,7 +135,7 @@ export class NestedStack extends Stack {
    * - If this is referenced from the parent stack, it will return `{ "Ref": "LogicalIdOfNestedStackResource" }`.
    * - If this is referenced from the context of the nested stack, it will return `{ "Ref": "AWS::StackId" }`
    *
-   * @example arn:aws:cloudformation:us-east-2:123456789012:stack/mystack-mynestedstack-sggfrhxhum7w/f449b250-b969-11e0-a185-5081d0136786
+   * @example "arn:aws:cloudformation:us-east-2:123456789012:stack/mystack-mynestedstack-sggfrhxhum7w/f449b250-b969-11e0-a185-5081d0136786"
    * @attribute
    */
   public get stackId() {

packages/@aws-cdk/aws-cloudformation/package.json

@@ -101,10 +101,7 @@
       "construct-ctor:@aws-cdk/aws-cloudformation.PipelineCloudFormationDeployAction.<initializer>",
       "construct-ctor-props-optional:@aws-cdk/aws-cloudformation.AwsCustomResource",
       "no-unused-type:@aws-cdk/aws-cloudformation.CloudFormationCapabilities",
-      "props-physical-name:@aws-cdk/aws-cloudformation.CustomResourceProps",
-      "docs-public-apis:@aws-cdk/aws-cloudformation.CustomResourceProvider",
-      "docs-public-apis:@aws-cdk/aws-cloudformation.CustomResourceProvider.serviceToken",
-      "docs-public-apis:@aws-cdk/aws-cloudformation.NestedStackProps"
+      "props-physical-name:@aws-cdk/aws-cloudformation.CustomResourceProps"
     ]
   },
   "stability": "stable"

packages/@aws-cdk/aws-cloudformation/test/example.customresource.lit.ts

@@ -25,7 +25,7 @@ class CopyOperation extends Construct {
     });
 
     new CustomResource(this, 'Resource', {
-      provider: CustomResourceProvider.lambda(lambdaProvider),
+      provider: CustomResourceProvider.fromLambda(lambdaProvider),
       properties: {
         sourceBucketArn: props.sourceBucket.bucketArn,
         targetBucketArn: props.targetBucket.bucketArn,

packages/@aws-cdk/aws-cloudformation/test/integ.trivial-lambda-resource.ts

@@ -22,7 +22,7 @@ class DemoResource extends cdk.Construct {
     super(scope, id);
 
     const resource = new CustomResource(this, 'Resource', {
-      provider: CustomResourceProvider.lambda(new lambda.SingletonFunction(this, 'Singleton', {
+      provider: CustomResourceProvider.fromLambda(new lambda.SingletonFunction(this, 'Singleton', {
         uuid: 'f7d4f730-4ee1-11e8-9c2d-fa7ae01bbebc',
         // This makes the demo only work as top-level TypeScript program, but that's fine for now
         code: new lambda.InlineCode(fs.readFileSync('integ.trivial-lambda-provider.py', { encoding: 'utf-8' })),

packages/@aws-cdk/aws-cloudformation/test/test.resource.ts

@@ -146,7 +146,7 @@ export = testCase({
     const stack = new cdk.Stack(app, 'Test');
     new CustomResource(stack, 'MyCustomResource', {
       resourceType: 'Custom::MyCustomResourceType',
-      provider: CustomResourceProvider.topic(new sns.Topic(stack, 'Provider'))
+      provider: CustomResourceProvider.fromTopic(new sns.Topic(stack, 'Provider'))
     });
     expect(stack).to(haveResource('Custom::MyCustomResourceType'));
     test.done();
@@ -160,7 +160,7 @@ export = testCase({
       test.throws(() => {
         new CustomResource(stack, 'MyCustomResource', {
           resourceType: 'NoCustom::MyCustomResourceType',
-          provider: CustomResourceProvider.topic(new sns.Topic(stack, 'Provider'))
+          provider: CustomResourceProvider.fromTopic(new sns.Topic(stack, 'Provider'))
         });
       }, /Custom resource type must begin with "Custom::"/);
 
@@ -174,7 +174,7 @@ export = testCase({
       test.throws(() => {
         new CustomResource(stack, 'MyCustomResource', {
           resourceType: 'Custom::My Custom?ResourceType',
-          provider: CustomResourceProvider.topic(new sns.Topic(stack, 'Provider'))
+          provider: CustomResourceProvider.fromTopic(new sns.Topic(stack, 'Provider'))
         });
       }, /Custom resource type name can only include alphanumeric characters and/);
 
@@ -188,7 +188,7 @@ export = testCase({
       test.throws(() => {
         new CustomResource(stack, 'MyCustomResource', {
           resourceType: 'Custom::0123456789012345678901234567890123456789012345678901234567891',
-          provider: CustomResourceProvider.topic(new sns.Topic(stack, 'Provider'))
+          provider: CustomResourceProvider.fromTopic(new sns.Topic(stack, 'Provider'))
         });
       }, /Custom resource type length > 60/);
 
@@ -224,7 +224,7 @@ class TestCustomResource extends cdk.Construct {
 
     this.resource = new CustomResource(this, 'Resource', {
       ...opts,
-      provider: CustomResourceProvider.lambda(singletonLambda),
+      provider: CustomResourceProvider.fromLambda(singletonLambda),
     });
   }
 }

packages/@aws-cdk/custom-resources/.gitignore

@@ -3,6 +3,7 @@ tsconfig.json
 tslint.json
 *.js.map
 *.d.ts
+!lib/provider-framework/types.d.ts
 *.generated.ts
 dist
 lib/generated/resources.ts
@@ -15,4 +16,4 @@ coverage
 .LAST_PACKAGE
 *.snk
 
-lib/sdk-api-metadata.json
+lib/aws-custom-resource/sdk-api-metadata.json