Design : Controller
The Controller component is responsible for tracking which tables are being replicated, as well as handling the process of adding and removing replications. It consists of a DynamoDB table and six lambda functions.
The controller DynamoDB table holds entries for each replication, identified by the name of the replicated table. A table entry has 3 mandatory fields: tableName, step, and state. The step field indicates a replications current place in the replication process. It can hold one of the following 6 values:
- VALIDATE_SOURCE
- VALIDATE_REPLICA
- CREATE_REPLICA
- START_REPLICATION
- REPLICATING *
- STOP_REPLICATION
With the exception of REPLICATING, all of these steps are mapped to a lambda function which executes the required actions. Once a table is set up and replicating, it remains in the REPLICATING state until it is stopped or removed.
The state field indicates the status of the replications current step. It can be set to one of the following 5 values:
- PENDING
- IN_PROGRESS
- COMPLETE
- FAILED
- ACTIVE *
The REPLICATING step can only be in the ACTIVE state. It is also the only step that can enter this state.
Controller - [src]
The controller function will be invoked whenever an item is added, removed, or modified in the controller DynamoDB table, and acts depending on the value of it's current state.
If the state is PENDING, the controller will invoke the lambda function mapped to the current step. The replication item is updated depending on the function's response. The state is set to FAILED if the function responds with an error, or fails to respond. Otherwise, the state is set to COMPLETE. Additionally, any key/value pairs set in the response will be added to the replication item, providing a means for storing and communicating data between steps.
Note: A step function can explicitly set new step and/or state values in the response object.
If the state is COMPLETE, the controller will move the replication to the next step, also setting the state back to pending. This way, the controller will be invoked again to complete the next step. If there is no more steps, the controller will instead delete the item, as the replication process is deemed complete.
If the state is IN_PROGRESS, FAILED, or ACTIVE, the controller takes no action.
Validate Source - [src]
The validate source function checks to see if the source table is configured properly for replication.
To be considered valid, the source table match must:
- Exist
- Not have a global tag set to true (this is to avoid replicating global tables)
- Have streams enabled
- Have a stream view type of NEW_AND_OLD_IMAGES
If any of these conditions are not met, the function will return unsuccessful, failing the replication process.
Validate Replica - [src]
The validate replica function checks to see if the replica table already exists. If the table exists, it compares the source and replica key schemas.
If the replica table exists and the key schemas do not match, the function returns unsuccessful, failing the replication process. Otherwise, the function returns successful.
If the table exists with a matching schema, the controller will skip the CREATE_REPLICA step and proceed directly to START_REPLICATION.
Create Replica - [src]
The create replica function creates a new DynamoDB table in the replica region to match the source.
The replica table will have the same name, key schema, attribute definitions, provisioned throughput, stream specification, local secondary indexes, and global secondary indexes as the source table.
It will return successful unless there was a issue creating the table.
Note: The create replica function does not copy table data
Start Replication - [src]
The start replication adds an event source mapping for the source table to the Replicator function, allowing it to start reading from the stream and writing to the replica table. It will return successfully unless there was an issue adding the event source mapping, setting the step to REPLICATING and the state to ACTIVE.
Note: The replicator starts reading data from the oldest point in the stream (up to 24 hours), so data processed during the setup will not be missed.
Stop Replication - [src]
The stop replication step removes the event source mapping for the source table from the replicator function, stopping the replication of data. It will return successfully unless there was an issue removing the event source mapping.