This is a demo for serverless computing. The demo implements an event streaming / sourcing pattern using serverless components from AWS. The lock-in to AWS is controlled by the Serverless.com framework.
The provided solution queries over Twitter API tweets based on certain buzzwords (hashtags). This query inside a lambda is run every minute by a scheduler. Received tweets are packaged as events an put into a Kinesis stream. See ./poll-tweet/README.md for more.
These events are collected and aggregated by another lambda. An aggregate consist of the core buzzword and sum of occurrences of other buzzwords in the relevant tweets. The aggregates are again package as events an put into a second Kinesis stream. See ./collect-buzzwords/README.md for more.
The events with the aggregated values for latest batch are collected by a next lambda which makes final aggregation with the already persisted data for each buzzword in DynamoDB. See ./persist-aggregates/README.md for more.
The definition of the buzzwords for queries are stored as state in DynamoDB. See ./poll-tweet-state/README.md for more.
A specialized set of BFF (backend for frontend) lambdas allows an UI client to present data and update state if necessary. See ./bff-state-and-aggregate/README.md
The solution follows one of the core principles when building serverless applications named SoC (separation of concerns). Each lambda functions realize only one tasks. These tasks are grouped in modules described shortly below.
The Twitter API keys are managed as secrets by SecretManager and diagnostic information is made available using CloudWatch.
sudo -H pip install --upgrade awscli --ignore-installed six
aws kinesis create-stream --stream-name $AWS_EVENT_STREAM_NAME --shard-count 1
aws kinesis put-record --stream-name $AWS_EVENT_STREAM_NAME --partition-key 123 --data testdata
aws kinesis describe-stream --stream-name $AWS_EVENT_STREAM_NAME --region $AWS_REGION
SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name $AWS_EVENT_STREAM_NAME --query 'ShardIterator')
aws kinesis get-records --shard-iterator $SHARD_ITERATOR
aws kinesis delete-stream --stream-name $AWS_EVENT_STREAM_NAME
The following resources proved to be useful during the creation of this demo