New serverless pattern - kinesis-lambda-dynamodb-pipeline-dotnet-cdk

[tejasavora]

Issue #, if available:
N/A

Description of changes:
This pull request implements a new serverless pattern demonstrating a real-time data pipeline using Amazon Kinesis, AWS Lambda, Amazon DynamoDB, and Amazon SQS. The key changes include:

1. Created a new CDK stack (KinesisLambdaDynamoDbCdkStack) that sets up:

   • An Amazon Kinesis Data Stream for data ingestion
   • An AWS Lambda function for data processing
   • Two Amazon DynamoDB tables: one for processed data and another for error logging
   • An Amazon SQS queue as a Dead Letter Queue (DLQ) for handling persistently failing records

2. Implemented a Lambda function (DataProcessFunction) that:

   • Processes records from the Kinesis stream
   • Stores successfully processed data in DynamoDB
   • Logs processing errors to a separate DynamoDB table

3. Configured robust error handling:

   • Set up retry attempts for failed record processing
   • Implemented automatic forwarding of consistently failing records to the DLQ

4. Added comprehensive testing instructions, including:

   • Steps for testing with AWS CLI and console
   • Information on running local unit and integration tests
   • Suggestions for end-to-end testing using AWS Step Functions

5. Updated README.md with detailed information about the pattern, its architecture, deployment instructions, and testing procedures.

This pattern showcases best practices for building scalable, resilient serverless data processing pipelines using AWS services and the AWS CDK with .NET.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[bfreiberg]

Hi @tejasavora , thanks for your contribution. Our serverless patterns are designed to be primarily Infrastructure as Code (IaC) implementations that showcase 2-4 AWS services working together, with minimal custom code. The goal is to demonstrate commonly used combinations that help developers quickly get started with serverless architectures.

This submission presents more of a complete application as such it would not align with our focus. I would suggest to simplify the code.

Thank you

[tejasavora]

Hi @bfreiberg I have updated PR with relevant changes - Removed DataIngestionFunction, Removed DynamoDB table for errors, removed DLQ and have made proper changes to ReadMe, example-pattern and architecture diagram.