CloudFormationController - refactor CloudFormation stacks

[michanto]

Description

A set of constructs that allows for higher level manipulation of CloudFormation stacks.

Working Backwards

Announcing the Launch of the CloudFormationController suite of tools!

What is it?

The CloudFormationController is a set of CDK-constructs that assist in the process of refactoring your hand-written CloudFormation stacks into CDK stacks. It helps you convert your CFN templates to CDK! It can refactor your CloudFormation stacks! It can rename CloudFormation stacks! It can rename stack outputs! It can freeze stack outputs! It can import existing resources into your stack! And it does this all this from your code pipeline, a stage at a time, following a process that you can easily test on a developer account – replacing confusing and time-consuming manual processes with tested, working, code pipeline-deployed stack transformations. You can even run test scenarios in development accounts to ensure the transforms work.

The CloudFormationController is a must-have tool for anyone who deploys CDK stacks across multiple pipelines in multiple accounts, as well as anyone attempting to migrate from hand-crafted templates to infrastructure as code (CDK).

How it works

StackTransformer

The StackTransformer generates a sequence of CloudFormation stack templates that model the transitions your CloudFormation stacks will go through during the refactor. It refactors CloudFormation stacks by performing consecutive deployments of those templates.

The StackTransformer is a CloudFormation custom resource. It is a Lambda-based resource that runs a StepFunction. Constructs allowing you to write your own StepFunction-based custom resources are included with the StackTransformer.

The StackTransformer itself builds intermediate templates into your CDK package, allowing you to see exactly what templates will be deployed in your account. The StackTransformer packages those templates up as CDK Assets, and uses an S3 BucketDeployment to move the templates to an S3 bucket, from which they can be deployed.

export interface StackTransformerProps {
    readonly env: Required<Environment>
    readonly transforms: StackTransformation[]
}

export interface StackTransformation {
    /**
     * Only fill in this or {@link deleteStackInput}.
     *
     * Fill in the TemplateBody.  StackTransformer will package up the template
     * in a BucketDeployment, delete the TemplateBody field, and fill in the
     * TemplateURL field for you.
     */
    readonly createChangeSetInput?: CreateChangeSetInput

    /**
     * Only fill in this or {@link createChangeSetInput}.
     */
    readonly deleteStackInput?: DeleteStackInput
}

StackRenamer

The StackRenamer is a StackTransformer that takes a source stack (as a CDK stack or downloaded template) and a target stack (as a CDK stack). It exports the resources from the source stack, and imports them into the target stack. This enables the customer to change both the name of the stack, and the logical resource ID’s of resources in the stack.

To rename a stack, the StackTransformer takes a number of actions on your behalf:

1. Freezes output values on the source stack to their string values for any outputs that are being used, because we will be removing those references from the source stack. It also can remove outputs that are not being used.
2. Exports any resources we want to appear in the target stack.
   1. Sets their DeletionPolicy to RETAIN in one deployment.
   2. Removes them from the stack in a subsequent deployment
3. Imports any resources into the new stack.
4. Deploys the full template for the new stack.

The StackTransformer does all this work for you. It deploys steps 1+2.i, then 2.ii, then 3. That’s three CloudFormation deployments. After that, your code pipeline can perform step 4. The StackTransformer can pick up where it left off – it knows what step in the process it was on, and can resume from a partially-successful state. If the StackTransformer was previously successful, it will be a no-op on subsequent deployments.

You can customize which resources are moved via the ResourceDestinationMap. Example:

export enum ResourceDestination {
    KeepInSourceStack,
    MoveToTargetStack,
    RetainDetached,
    DeleteResource
}

static resourceDestinations: ResourceDestinationMap = {
    // BucketNotMovingId will not be moved.
    "BucketNotMovingId": ResourceDestination.KeepInSourceStack,
    // Move every S3 bucket, except those with overrides in this map.
    "AWS::S3::Bucket": ResourceDestination.MoveToTargetStack
}

You can change their logical ID:

static resourceIdChanges: ResourceIdChanges = {
    "BucketThatIsMovingId": "NewIdForBucketThatIsMoving",
}

Or accept the default to move as much as possible as-is.

export interface StackRenameTransformsProps extends StackTransformationsProps {
    /**
     * The current source stack template
     */
    readonly sourceStack: ITransformStack
    /**
     * The desired target stack template, when this process is done.
     * Note:  The existing target stack template will be downloaded from
     * the account if necessary.
     */
    readonly targetStack: ITransformStack

    /**
     * List of resources from the source stack and their destinations.
     * Every resource in the Source stack needs to have a destination.
     * Sources can be listed by Type or by source stack ID.
     *
     * For example, you can specify that all Buckets except specific ones are moved.
     */
    readonly resourceDestinations: ResourceDestinationMap

    /**
     * This list maps a source stack resource id to a target stack resource id.
     * If the resource id is the same in both stacks, it does not need to be listed here.
     *
     * Note that this list is ONLY for mapping source stack IDs to target stack IDs.
     * It does not control which resources will move.  If a resource is listed here, but not in
     * resourceDestinations (either by ID or type), it will not move.
     */
    readonly resourceIdChanges: ResourceIdChanges
}

export interface ITransformStack {
    readonly template?: any
    readonly parameters?: Parameters
    readonly stackName: string
}

ResourceImporter

Have you ever wanted to add an alarm to a logGroup? But you’ve deployed this template to several accounts, and some of them have the logGroup (e.g. because the lambda has run), and some of them don’t (because that lambda has not run in that account yet). What do you do? You could use LogGroup.fromLogGroupName(), but you don’t know which accounts the LogGroup exists in and which it doesn’t – and LogGroup.fromLogGroupName doesn’t know either! The ResourceImporter is here to help – it can detect if a resource exists in an account, if the resource is in the template for that account already, and if not, it can generate a template and import that resource for you. After the ResourceImporter runs, run your normal stack deployment to deploy any resources dependent on the newly imported LogGroup.

To use the ResourceImporter, simply indicate the old stack template, a list of resources you wish to import, and the new stack template. Don’t worry! We can download the old stack template for you if you don’t have it handy. It will be downloaded to the context directory in your CDK package, and you check it in along with your code.

Reference: https://www.endoflineblog.com/cdk-tips-03-how-to-unblock-cross-stack-references

ReferenceResolver

The ReferenceResolver is a StackTransformer that can change the outputs in one stack, and fix up the import in the second stack (or stacks) for you. The ReferenceResolver takes as input the Source stack, the Target stacks for the export, and the old and new names for the Output(s).

The ReferenceResolver should be deployed before the new Source and Target stacks: that way it can deploy the existing templates for the Source and Target stack with the output modifications. This is accomplished in three steps:

1. Deploy the Source stack with both the old and new names.
2. Deploy the Target stacks with the new names.
3. Deploy the Source stack with the new names only.

Now your pipeline (or you) can deploy the Source stack (which probably has not changed from deployment 3 above), followed by the Target stacks (which probably haven’t changed from deployment 2 above). When run again, the ReferenceResolver detects that the new values are in place and does nothing.

You can provide the existing templates as Stack constructs, and the StackTransformer will make the edits for you, or the StackTransformer can download them at build time (you will have to check them in, they are downloaded under the context directory in your package root).

OutputFreezer

The OutputFreezer is a StackTransformer that can freeze the values of any in-use outputs in a stack to their string equivalents. This allows you to remove the resources from the stack while keeping the outputs in place for any consumers.

CfnStack

This Stack class wraps (and can even download) your hand-crafted CFN stack and wrap it in a CDK Stack that you can deploy. It handles parameter substitution, lambda asset packaging, and various other CloudFormation transforms. It allows the developer to alter the resulting CloudFormation in code - allowing you to continue deploying your existing CFN template or the CfnStack.

AccountContextProvider

Run any aws command and use the output as part of the logic for your CDK stack. The AccountContextProvider will run your AWS command as part of your build, cache the response on-disk for use on the build machine, and return the response to you. You can model your AWS command either as a cli command or an SDK call.

Want to know if a LogGroup exists in an account?
Do you need the template of a stack?
Do you need to list stack outputs, resources, or parameters for your CDK logic?

The AccountContextProvider can perform any AWS CLI or SDK command and cache the result as part of your build. Create the cache on a developer machine, and check it in. That snapshot of account state can be used to decide which resources to import to your stack, such as whether you need to create or import a LogGroup for the Alarm you are adding.

StepFunction Based Custom Resources

The StackTransformer is a StepFunctionCustomResource. This is a lambda-based custom resource that calls a StepFunction. The StepFunctionCustomResource ensures that your resource will respond to CF even if a step function lambda calls fails. This allows you to automate any process you can call from a StepFunction, right from your CloudFormation stack!

Roles

• Driver (drives the proposal to completion): @michanto
• Approver(s): (assigned by CDK team)

[eladb]

Hi @michanto, this looks very interesting. We are in the process of revising our RFC workflow.

Please take a look at the new workflow as described here:
https://github.com/aws/aws-cdk-rfcs/tree/benisrae/rfc-2#rfc-process

[michanto]

I think the above functionality breaks down into the following packages:

cdk-patterns

Use the same patterns the CDK does to extend the Construct Tree!

Internally, the AWS CDK attaches extra data and properties to the construct tree. Then the CDK uses that data to assist in generating CloudFormation. Now you can too! CDK Patterns gives you the ability to store arbitrary data on the construct tree that can be applied at different points during the CDK App lifecycle.

With CDK Patterns you can:

• Late-bind construct or resource properties.
• Remove dependencies between constructs.
• Modify CloudFormation at different points in the CDK App lifecycle.
  • Hook into existing methods to modify your CloudFormation output.
  • Manage Aspect hosts and data.
• Use the Construct tree as an IOC container for dependency-injection.

CDK Patterns is THE toolkit for creating advanced CDK constructs.

cdk-transforms

The CDK Transforms library brings CloudFormation transform macros to the CDK.

CDK Transforms can alter CloudFormation at different points in the CDK app lifecycle:

• Before it is imported into a CfnInclude construct.
• Between when it's generated by the stack and when it's written to cdk.out.

You can remove or replace any resource, change any property, alter mappings – all at the CloudFormation level. It's the ultimate escape hatch for CloudFormation generation! Target your changes to part or all of the template. If you can't do it in the CDK, but you can do it in CloudFormation, CDK Transforms gives you a hook to make it happen. Easily installed in any CDK Stack or CfnElement construct you can subclass.

CDK Transforms can help you import tool-generated or hand crafted CloudFormation without altering the producing tool. Your existing CloudFormation becomes part of your CDK build, transformed into a CDK-friendly format. Continue deploying CloudFormation with your existing process while the CDK conversion is ongoing. No need to stop existing development- go at your own pace and transition when you're ready. Reap the benefits of the CDK in new accounts while using your legacy code base in existing accounts. Dual-deploy until you are ready to sunset your old process.

cdk-context

CDK Transforms can do anything a CloudFormation Transform Macro can do...except call the AWS SDK. CDK Context fills that gap by allowing you to call any AWS SDK function on a developer box, then check in the resulting account context data for use in official builds on credential-less build machines.

• Download existing Stack templates and make them part of your CDK build!
• Answer questions such as:
  • Is this stack deployed into the account?
  • What is the physical id of this resource?
  • Does this log group exist?
    • Is the log group in the stack?
    • Do I need to create/import the log group before adding alarms?
  • Can I change a Stack output name?
    • It this Stack output being used?
    • By which other stacks?
  • What is the current stack drift?
    • Can I resolve the current stack drift?

You can use the answers to these questions to determine what your template includes, making it possible to manage installations of the "same" CloudFormation stack across multiple accounts. Reason about the differences between the current stack template, the desired stack template, and the current actual state of the account. You can even validate current account state before deploying your CloudFormation changes to ensure the context data you relied on is not stale - stopping a potentially complicating deployment before it happens.

cdk-refactor

CDK Refactor combines CDK Transforms and CDK Context to create a powerful CloudFormation stack refactoring mechanism.

CDK refactor can:

• Move resources between stacks.
• Import existing resources into your stack.
• Help you disentangle the dependencies between resources – or between stacks.
• Help you resolve drift between the template and the resources.
• Help you resolve drift between accounts deployed with the "same" CloudFormation stack.

CDK refactor is a must-have tool for anyone managing deployments of the "same" CloudFormation stack across multiple disconnected (platform) or connected (cellular) accounts.

[awsmjs]

Closing this ticket as it does not reflect current priorities. We don't have the bandwidth to collaborate on design or implementation. We suggest you pursue experimentation in a separate package or a fork if needed. If a successful implementation emerges, reopen the proposal with details on the functionality and how it can be implemented in the core library.