[T23.00] Step Functions Orchestration building e2e data pipeline using step functions.md

[T23.00] Step Functions Orchestration: building e2e data pipeline using step functions

Documentation:

• AWS Step Functions Overview
• What are AWS Step Functions?
• Choice
• Wait
• AWS::StepFunctions::StateMachine

Description:

Create State Machine in AWS StepFunctions service UI with the same Job Flow for dim_go_methods, dim_time_period. The workflow is started manually

Files for pushing to CodeComet:

• CloudFormation template: {user_id}_dim_go_methods_t17.yaml, {user_id}_dim_time_period_t17.yaml

Amazon Development:

1. Create a State Machine on cluster with the name: {user_id}_dim_go_methods_t17, {user_id}_dim_time_period_t17
2. Job Flow:

raw --> stage --> analytics --> start_crawler --> get_crawler_state -->|-- if crawler is not in ready state  --| --  |-- else --| --> datawarehouse
                                                                       |--        wait for 30 sec            --|
                                                          ^^^--------------------------   

3. Create AWS CloudFormation template for State Machines with the name: {user_id}_dim_go_methods_t17.yaml, {user_id}_dim_time_period_t17.yaml
4. Test CloudFormation Template:
   1. Deploy (check that your State Machines has been created using CloudFormation)
   2. Delete your CloudFormation Stack
5. Open Pull Request as described in Common Info (Task Workflow) section.