aws-step-functions-ebs-snapshot-mgmt

Example architecture for integrating AWS Step Functions and Amazon CloudWatch Events.

The Snapshot Management Application (diagram) demonstrates how to use AWS Step Functions in conjunction with Amazon CloudWatch Events, AWS Lambda, and AWS CloudFormation to build a serverless solution for Amazon Elastic Block Store (EBS) snapshot lifecycle management.

The application assumes that you already are using something to schedule the creation of snapshots for your EBS volumes, and the reference architecture initiates once those snapshots are completed. We'll setup an Amazon CloudWatch Event that will trigger on the completion of the snapshot creation. The target for the CloudWatch event is an AWS Step Functions state machine. The state machine coordinates different steps in the EBS snapshot management, including deleting snapshots past the retention period specified, and copying snapshots to a Disaster Recovery (DR) region. We deploy another state machine in the DR region that performs similar steps for the snapshots that are copied into the DR region.

This repository contains sample code for all the AWS Lambda functions that the AWS Step Functions state machines invoke, and AWS Serverless Application Model (SAM) templates for deploying the Lambda functions and the state machines. The Amazon CloudWatch Events rules matching EBS snapshot events to target state machines can be created using a manual creation via the CloudWatch Events console or the AWS Command Line Interface (CLI).

Deploying the example

Because the example involves resources in two regions, the primary region and DR region, there are two stacks that are needed to launch the application.

The first stack is executed in the primary region. The Launch Stack button below will launch the template for the primary region in the eu-west-1 (Ireland) region in your account:

The second stack is executed in the DR region. The Launch Stack button below will launch the template for the DR region in the us-east-2 (Ohio) region in your account:

After the stack is successfully created, you can test the configuration by following the instructions in the Testing the example section.

Configuration Options

There are a few different configuration options for controlling the Snapshot Management architecture.

1. Configure Notifications for Failures: Whenever a failure is detected in the state machine execution (or AWS Lambda functions it invokes) the state machine executes a Lambda function that sends a notification to an Amazon Simple Notification Service (SNS) topic. To receive emails when a failure occurs, add an email subscription to the SnapshotMgmtTopic.

2. Configure Volumes to Include: If you only want certain volumes to be included in the snapshot management workflow, you can specify a Tag key that a volume must have in order for it to be included. If a Tag key isn't specified, then the snapshot management will take place for all snapshot creations. If you would like to specify a Tag key, either:

• After the CloudFormation stack has completed in your primary region. Follow these steps to modify it:
  • Go to Services -> Lambda
  • Select the TagSnapshots function
  • On the Code tab, scroll to the bottom and in the Environment Variables section, fill in the tagKey environment variable with the value of your tag key you want to perform snapshot management for.
• Edit the PrimaryRegionTemplate.yaml prior to deployment (if you are following the steps in the section that describes How to customize and run the architecture in your account). You will modify the tagKeyValue Default value in that file.

tagKeyValue:
  Description: 'The value for the key tag that you want all volumes to have for the snapshot management to apply.'
  Type: 'String'
  Default: 'none'

Testing the example

The application can be tested by performing the following steps:

1. Login to the AWS Management Console.
2. In the upper right hand corner, choose the primary region (Ireland).
3. From the Services menu in the top left, choose Amazon EC2.
4. Click Volumes from the menu on the left side.
5. If you do not already have a volume, create a volume.
6. Select the volume you want to snapshot, select Create Snapshot from the Actions drop-down menu.

This will start the state machine. You can see the completion of the state machine by choosing AWS Step Functions from the Services menu. Choose the SnapshotMgmtStateMachine- from the list of state machines. This displays a list of executions. Click on an execution to see the details of the state machine execution. You can also switch to the DR region (Ohio) in the upper right corner and see the execution of the DR region state machine.

Cleaning up the example resources

To remove all resources created by this example, do the following:

1. Delete the AWS CloudFormation stacks in the primary and DR regions.

How to customize and run the architecture in your account

Note: This assumes you have the AWS CLI installed and configured

First clone the repo:

# Clone it from github
git clone https://github.com/awslabs/serverless-stepfunctions-ebs-snapshots.git

Make the edits you want to make. For instance, if you want to modify the DR region (i.e. not use Ohio), then in the PrimaryRegionTemplate.yaml file, edit the default value for the DRRegion parameter to the region you would prefer to use:

DRRegion:
  Description: 'The DR region where snapshots will be copied (This should be a different region from the region you are running this CloudFormation stack in.'
  Type: 'String'
  Default: 'us-east-2'

In the following commands you'll need to replace the following: <PRIMARY_REGION> - Replace with primary region (i.e. us-east-1, etc.) <DR_REGION> - Replace with DR region (i.e. us-east-1, etc.) <PRIMARY_UNIQUE_BUCKET_NAME> - Replace with a globally unique bucket name for staging code in your primary region <DR_UNIQUE_BUCKET_NAME> - Replace with a globally unique bucket name for staging code in your DR region

Also, ensure that you have the latest CLI installed because the updates for the AWS Serverless Application Model (SAM) are needed for the next section.

These first commands create an Amazon S3 bucket for staging your AWS Lambda function zips, then package the code and upload it, then deploy the AWS CloudFormation stack.

# Create an S3 bucket for staging your code in the primary region
aws s3api create-bucket --bucket <PRIMARY_UNIQUE_BUCKET_NAME> --region <PRIMARY_REGION> --create-bucket-configuration LocationConstraint=<PRIMARY_REGION>
# If your primary region is us-east-1 run this command instead
aws s3api create-bucket --bucket <PRIMARY_UNIQUE_BUCKET_NAME> --region <PRIMARY_REGION>

aws cloudformation package --template-file PrimaryRegionTemplate.yaml --s3-bucket <PRIMARY_UNIQUE_BUCKET_NAME> --output-template-file tempPrimary.yaml --region <PRIMARY_REGION>

aws cloudformation deploy --template-file tempPrimary.yaml --stack-name PrimaryRegionSnapshotManagement --capabilities CAPABILITY_IAM --region <PRIMARY_REGION>

These next set of commands perform the same actions for the DR region.

# Create an S3 bucket for staging your code in the primary region by running
aws s3api create-bucket --bucket <DR_UNIQUE_BUCKET_NAME> --region <DR_REGION> --create-bucket-configuration LocationConstraint=<DR_REGION>
# If your DR region is us-east-1 run this command instead
aws s3api create-bucket --bucket <DR_UNIQUE_BUCKET_NAME> --region <DR_REGION>

aws cloudformation package --template-file DR_RegionTemplate.yaml --s3-bucket <DR_UNIQUE_BUCKET_NAME> --output-template-file tempDR.yaml --region <DR_REGION>

aws cloudformation deploy --template-file tempDR.yaml --stack-name DRRegionSnapshotManagement --capabilities CAPABILITY_IAM --region <DR_REGION>

At this point the stacks will be updated and you can begin creating snapshots.