Talkie App Backend

talkie App backend server is an interesting real-time social network application. It is developed using node.js, typescript, redis and mongodb.

You can find the repo for the frontend built with react here.

Features

1. Signup and signin authentication
2. Forgot password and reset password
3. Change password when logged in
4. Create, read, update and delete posts
5. Post reactions
6. Comments
7. Followers, following, block and unblock
8. Private chat messaging with text, images, gifs, and reactions
9. Image upload
10. In-app notification and email notification

Main tools

• Node.js
• Typescript
• MongoDB
• Mongoose
• Redis
• Express
• Bull
• PM2
• AWS
• Terraform
• Nodemailer
• Sendgrid mail
• Cloudinary
• Jest
• Lodash
• Socket.io

Requirements

• Node 16.x or higher
• Redis (https://redis.io/download/)
• MongoDB (https://www.mongodb.com/docs/manual/administration/install-community/)
• Typescript
• API key, secret and cloud name from cloudinary https://cloudinary.com/
• Local email sender and password https://ethereal.email/

You'll need to copy the contents of .env.development.example, add to .env file and update with the necessary information.

Local Installation

• There are three different branches develop, staging and main. The develop branch is the default branch.

git clone -b develop https://github.com/akibrahimug/talkie-backend.git
cd talkie-backend
npm install

• To start the server after installation, run

npm run dev

• Inside the setupServer.ts file, comment the line sameSite: 'none'.
• You'll need to uncomment that line again before you deploy to AWS.

Make sure mongodb and redis are both running on your local machine.

Unit tests

• You can run the command npm run test to execute the unit tests added to the features controllers.

API Endpoints

• The actual endpoints for the application can be found inside the folder named endpoints https://github.com/kasomaibrahim/talkie-backend/tree/develop/endpoints.
• The endpoint files all have a .http extension.
• To use this files to make api calls, install the extension called rest client on vscode.
• Update the endpoints http files before using.
• The files inside the endpoints folder contains APIs for
  • Authentication
  • Chat
  • Comments
  • Followers, following, block and unblock
  • Health
  • Images
  • Notification
  • Posts
  • Reactions
  • User

View Data

• You can view the contents of your redis cache by using a tool called redis-commander.
• Download mongodb compass to view data. https://www.mongodb.com/try/download/compass.

AWS Setup

• You can create an account on AWS if you don't have one already.
• Download and install aws cli.
• On AWS, create an IAM user. You'll get a key and secret.
• Use aws configure command to add your iam secret and key to your local machine.
• To deploy the backend server on AWS, it is required you have a domain to use.
• With that domain, manually create a route53 hosted zone on AWS.
• Copy the hosted zone NS properties and add to the nameservers section of your domain on the dashboard of your domain name provider. e.g: godaddy, namecheap etc.
• For example, on namecheap.com

AWS Resources Used

• VPC
• Subnets
• Internet gateways
• Route tables
• Elastic ips
• Nat gateways
• Security groups
• ALB target groups
• Route53
• AWS Certificate Manager
• Application load balancers
• IAM roles
• Elasticache
• EC2 launch config
• EC2 instances
• Autoscaling group
• S3
• Code deploy
• Cloudwatch

AWS Infrastructure Setup with Terraform

• Make sure to uncomment this line sameSite: 'none' inside setupServer.ts file.
• Install terraform
• Terraform stores state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures.
• This state is stored by default in a local file named "terraform.tfstate", but it can also be stored remotely, which works better in a team environment.
• Update the variables.tf file with the correct data. Update the properties with comments.
• To store your terraform remote state on AWS, first create a unique S3 bucket with a sub-folder name called develop.
• Add that S3 bucket name to main.tf file. Also add your region to the file.
• Create a keyPair on AWS. Keep the key safe on your local machine and add the name of the keyPair to ec2_launch_config.tf and bastion_hosts.tf files.
• Before running the terraform apply command to your resources, you need to
  • create a new s3 bucket to store env files
  • inside the created s3 bucket, add a sub-folder called backend and inside the backend folder another sub-folder called develop. Bucket path should be something like <your-s3-bucket>/backend/develop
  • in your project, create a .env.develop file. Add the contents of .env to the new file.
  • the contents of .env.develop needs to be properties that will be used when deployed
    • DATABASE_URL should be an actual mongodb url. You can create an account on mongodb atlas and create a new database.
    • NODE_ENV can be set to production
    • CLIENT_URL can be set to the frontend application local url or actual domain
    • API_URL should be https://api.dev.<your-backend-domain> that you specified inside your terraform variables.tf file.
    • SENDGRID_API_KEY and SENDGRID_SENDER should be created from sendgrid dashboard. Create an account.
  • after updating the .env.develop file, you can zip it and upload it to the new s3 bucket you created to store env files. Upload using aws cli
    • zip env-file.zip .env.develop. The file must be called env-file.zip.
    • aws --region <your-region> s3 cp env-file.zip s3://<your-s3-bucket>/backend/develop/
• Go into the user-data.sh and update-env-file.sh files and update.
• Once the env-file.zip has been added to your s3 bucket, you can execute inside the deployment folder, the commands:
  • terraform init
  • terraform validate
  • terraform fmt
  • terraform plan
  • terraform apply -auto-approve
• It will take sometime to create the resources. If everything works well, you should be able to access the backend dev api endpoints.
• To destroy all created resources, run
  • terraform destroy

Setup CI/CD Pipeline with CircleCI

• Create an account on circleci.
• Signup or login with the github or bitbucket account where you stored your code.
• Setup your project.
• Add the environment variables you see on the screenshot to circleci.
• For CODECOV_TOKEN, visit CodeCov and signup or login with the same account where you have your project stored.
  • Once you login and setup your project, you will receive a token. Add that token to circleci.
• For CODE_DEPLOY_UPDATE, the default value is false. That is the first time you are setting up your pipeline and infrastructure. Once everything is setup and running, if you need to update your backend code, then you can change the property to true.
• SLACK_ACCESS_TOKEN and SLACK_DEFAULT_CHANNEL can be obtained by following this documentation.

Before Starting a Build

• Inside the circleci.yml file, you need to make some updates.
• There are some properties named <variable-prefix> that you need to replace with the prefix value from your terraform variables.tf. Search the config.yml file and replace <variable-prefix>.

You can find the frontend code here

//Tools

• Node js with Express and Typescript => Runtime

• MongoDB => Database

• Redis => In memory cache

• Message Queue => Redis based message que

• Terraform => Infrastructure as code tool

• CircleCI => CI/CD platform

• Git => Version Control

• GitHub => Code hosting

• AWS => Cloud computing platform -- Compute - EC2 - Elastic Load Balancing -- Storage - S3 - CloudFront - Elastic Block Storage -- Developer - CodeDeploy

• Banyan => For logging errors in production

• Cloudinary => For deploying images

• Ethereal => Email brocker

• PM2 => Node Process Manager

  //App Features

• Authentication => Using Cookies (JWT) Signup signIn Signout Password Reset /Password request Current User Check on the frontend to make sure their token allows access

• Chat/Messaging Private chat Send Images in chat Add Message Reactions Retrive messages Mark as read Delete Messages

• User Get single user Get Multiple users with Pagination Select Random user to be followed

• Post Create post with and without images Get Posts Update post Edit posts

• Comments Add comments Get single and multiple comments

• Reactions Add Post reaction Get single and multiple post reaction Remove reaction from post

• Images Add Image to post Upload profile/background images Retrive images Delete profile and background image

• Folloewr/Unfollow/Block and Unblock Follow User Unfollow user Block and unblock user

• Notification Notification settings Retrive and Display notifications Delete and update notifications

The app is set up in a way that in future we can use it as a microservice

AWS SERVICES USED

• Virtual Private Cloud (VPC) It separates the public cloud(aws) to have a private cloud.
• Subnet Allows us to manage internet traffic into our instances IP addresses that we can use
• Internet GateWay Allows internet into our VPC
• Route Table Allow traffic within the VPC
  • Traffic through instances launched from either the private/public subnet
• Elastic IP Used with NAT Gateways Assign to EC2 instances
• NAT Gateways Launched from the public subnets giving access to the private subnets
• Bastion Host Allow admin access from the public subnet into the EC2/ElastiCache in the private subnet
• Security groups Firewalls for instances
• Application Load Balancers (ALB) Used with the Route 53 Balance the traffic entering into our servers
• Route 53 Use the nameserves to connect to external domains
• AWS Certificate Manager(ACM) Allow the app to only use https
• Auto Scaling Groups (ASG) Allows us to scale up or down the instances
• Elastic Compute Cloud (EC2) Computer on which the serve is running on in the cloud
• Elasticache We are using this to store the redis data
• IAM Roles and Policies Access temporaliy
• Simple Storage Service(s3) For storage
• CodeDeploy If we update our code base in GH this will help keep the app upto date