Skip to content

Latest commit

 

History

History
327 lines (230 loc) · 16.4 KB

README.md

File metadata and controls

327 lines (230 loc) · 16.4 KB
Raw

peblio screenshot

Thank you for wanting to contribute to Peblio. In this document, you will find instructions for the below -

Setup on M1 Mac:

  1. Remix this repository.
  2. Clone the forked repository and cd into it.
  3. Create a Python 3.8.11 environment and activate it.
  4. pip install -r requirements.txt --use-deprecated=backtrack-on-build-failures
  5. Make sure that you have Node.js v8.7.0 or higher installed.
  6. npm install
  7. cd client && npm install
  8. cd ../server && npm install
  9. Install MongoDB and make sure it is running.
    • For Mac OSX with homebrew: brew install mongodb then brew services start mongodb
  10. Get the AWS IAM credentials for the peblio-local-development user from Mathura and place them in your ~/.aws/credentials file under a profile called peblio. You should also create a peblio profile in ~/.aws/config with the line region=us-east-1. The easiest way to do this is to run aws configure --profile peblio. See the AWS docs for more details.
  11. Get the Peblios.pem SSH key from Mathura and place it in your ~/.ssh directory. Then, if you haven't already, generate an SSH key for your GitHub account. Finally, add the following lines to your ~/.bashrc or ~/.bash_profile startup script: 
    ssh-add ~/.ssh/Peblios.pem
ssh-add ~/.ssh/id_rsa
    Alternatively, if you're using OS X, instead of modifying your ~/.bashrc, you can configure SSH to use the OS X Keychain to automatically make your private keys available to SSH.
  12. (Optional) Install the React Developer Tools and Redux Developer Tools Chrome extensions.
  13. In case you see error for not being able to load libffi brew install libffi
  14. Find where libffi.8.dylib is and run cp libffi.8.dylib libffi.7.dylib

Setup

  1. Remix this repository.
  2. Clone the forked repository and cd into it.
  3. (Optional) Create a Python 3.7.11 virtualenv and activate it.
  4. pip3 install -r requirements.txt
  5. Make sure that you have Node.js v16.20.2 or higher installed.
  6. npm install
  7. cd client && npm install
  8. cd ../server && npm install
  9. Install MongoDB and make sure it is running.
  10. Get the AWS IAM credentials for the peblio-local-development user from Mathura and place them in your ~/.aws/credentials file under a profile called peblio. You should also create a peblio profile in ~/.aws/config with the line region=us-east-1. The easiest way to do this is to run aws configure --profile peblio. See the AWS docs for more details.
  11. Get the Peblios.pem SSH key from Mathura and place it in your ~/.ssh directory. Then, if you haven't already, generate an SSH key for your GitHub account. Finally, add the following lines to your ~/.bashrc or ~/.bash_profile startup script: 
    ssh-add ~/.ssh/Peblios.pem
ssh-add ~/.ssh/id_rsa
    Alternatively, if you're using OS X, instead of modifying your ~/.bashrc, you can configure SSH to use the OS X Keychain to automatically make your private keys available to SSH.
  12. (Optional) Install the React Developer Tools and Redux Developer Tools Chrome extensions.

Local Development

  1. cd client && npm start
  2. In another terminal session, cd server && npm start
  3. Navigate to http://localhost:8080 in your browser.

Nodeman has been configured to do life refresh on the server side. After any changes to the server, wait for the files to get re-interpreted and type rs on the console where the server runs

StoryBook

We have storybook installed for developing and designing basic components. To start storybook -

  1. cd client && npm run storybook
  2. Your browser should automatically open the Storybook UI

Currently, the storybook files are present in client/src/components/**/*/.stories.js We can use this as a template to build further components

Local Server Debug

Please note, the debug only works in Visual Studio Code. Also the debug does not work with python being used in virtualenv.

  1. Install Visual Studio Code and open project
  2. The launch.json and tasks.json shall create a Debug Launch Confirguration "Server Debug"
  3. Install a breakpoint and start the Debug

The presence of jsconfig.json file in a directory indicates that the directory is the root of a JavaScript Project. It also helps us point that server is ES6 syntax javascript project

Babel in Server

To be able to use ES6 syntax, we use babel on the server side. As mentioned in babel-node docs, You should not be using babel-node in production. It is unnecessarily heavy, with high memory usage due to the cache being stored in memory. You will also always experience a startup performance penalty as the entire app needs to be compiled on the fly. As an option, while launching the server on higher environments we use npm run startserver. With this, we first transpile code to backwards compatible version of JavaScript and then start server. This is done in the prestartserver script of node.

While launching locally, we use npm start which calls prestart script from package.json.

The reason we have different server startup scripts for local and for higher environments is because npm start generates source-maps which helps in local debugging. This isn't required while starting the server on higher environment.

Tests

This project uses TestCafe to run end-to-end tests against the frontend and backend.

You can run the tests once by running

npm test

You can run the tests in watch mode (using TestCafe Live) by running

npm run test:watch

By default, these commands run the tests using a headless Firefox browser. However, there are several other test commands defined in package.json that will run the tests against other browsers. For example, to run the tests against Chrome, Firefox, and Safari simultaneously, you could run

npm run test:all

Unit Tests For Server

You can run the from root folder by running

cd server
npm test

Unit Tests For Client

We have used enzyme to test react components along with chai, mocha, sinon You can run the from root folder by running

cd client
npm test

Integaration Tests For Server

We have used supertest to write integration tests along with chai, mocha, sinon. The framework brings up the server and uses ENVIRONMENT local configuration You can run the from root folder by running

cd server
npm run integrationTest

Git Workflow

The expected git workflow for feature development is:

  1. Either create a local branch for your feature, OR, create a branch for your feature in your fork of the repo.

  2. When your work is ready, create a pull request against the master branch of this repo.

  3. Once your pull request to master has been merged, create a pull request from master to staging, merge it, and deploy to the staging environment to manually test your feature.

  4. Once you've verified that everything works in the staging environment, you can create a pull request from staging to production, merge it, and deploy to the production environment.

Deploying

Backend

The backend uses ansible to deploy code to EC2 servers. For more info on how this works, see the Ansible section of this README.

Staging

cd server
./devops/staging_deploy.sh

Production

cd server
./devops/prod_deploy.sh

Frontend

The frontend uses the AWS CLI to push the latest build to S3 and invalidate the corresponding CloudFront cache.

Staging

git checkout staging
cd client
./devops/staging_deploy.sh

Production

git checkout production
cd client
./devops/prod_deploy.sh

Credstash and Environment Variables

Backend

On the backend, Peblio uses credstash, a utility that uses AWS KMS to securely store data in AWS DynamoDB, to manage secrets that should not be checked into version control.

server/run_with_credstash.sh takes care of mapping credstash secrets to environment variables used by the Express server. This bash script expects you to have an AWS profile called peblio configured with the credentials for the peblio-local-development IAM user.

Listing Secrets

To see a list of the secrets currently stored in credstash:

cd server
./list_credstash_secrets.sh

Adding a Secret

To add a secret to credstash:

cd server
./add_credstash_secret.sh my.secret donttellanyone local

When adding a new secret, always make sure to add a version for each of the following environments: local, test, staging, and production.

Once you've added a secret, make sure that you also update server/run_with_credstash.sh to map that secret to an environment variable.

Deleting a Secret

If you need to a delete a secret, you can run:

cd server
./delete_credstash_secret.sh my.secret local

However, DO NOT DO THIS unless you're sure that it won't affect the production servers or anyone else's work.

Frontend

Environment variables for the frontend are defined in three .env files in the client directory: .env, .env.staging, and .env.production.

These files are checked into version control, because any environment variable used in the frontend will be visible in the compiled JavaScript code. With that in mind, DO NOT PUT SENSITIVE DATA OR CREDENTIALS IN THESE FILES! Anything that needs to be kept secret should be stored in credstash and used exclusively on the backend.

Ansible

ansible is an IT automation tool that is used in this project to automate the creation, provisioning, and deployment of backend EC2 instances.

The structure of the server/devops/ansible directory roughly follows the recommended strategy for multistage environments described in this tutorial.

Inventories

Inventories specify which EC2 servers belong to which environments.

There are two separate inventories located in server/devops/ansible/inventories, one for staging and one for production. Both inventories use an EC2 Dynamic Inventory to find any currently running EC2 instances that belong to the corresponding environment.

The code that defines these two inventories is almost identical, with the key difference being a filter in the respective ec2.ini files that only includes EC2 instances with an env tag corresponding to the environment.

Each inventory also defines group variables that are unique to that environment in various YAML files. These variables are used in roles when creating, provisioning, and deploying EC2 instances. Any variables that are invariant across environments are defined in server/devops/ansible/inventories/cross_env_vars.yml, which is symlinked into both the staging and production inventories.

Roles

Roles define sets of tasks that are run either locally or remotely on EC2 instances via SSH. Some roles may also define their own variables or include template files that will be copied over to an EC2 instance.

Most roles are intended to be idempotent. That means that there should be no difference between running a role like deploy_webserver once vs. running in many times in a row. However, not all roles are idempotent - create_webserver spins up a new EC2 instance each time that it runs.

Roles don't specify which hosts they should run on - that's taken care of by playbooks.

Playbooks

Playbooks map host groups to roles. This allows the same role to be used multiple times for different hosts in different contexts, e.g. the deploy_webserver role is used in both create_webserver.yml and deploy_webserver.yml.

Bash Scripts

For convenience, there are several bash scripts (staging_deploy.sh, prod_deploy.sh, etc.) located in the devops directory that abstract away the details of running ansible for common tasks like deployment and provisioning.

AWS Profile

We use peblio profile for AWS and this is defined in AWS_PROFILE environment variable

Creating a new instance (front and back) - WIP

Let's talk about front end first

  1. Create an S3 bucket - this needs to be done manually
  2. When creating bucket, you can copy the settings from an existing bucket
  3. Setup static website hosting for the bucket once its created
  4. Create cloudfront - choose web distribution
  5. Make note of the distribution ID
  6. Update distribution ID and S3 bucket name in deploy.sh
  7. Post this, you can use the deploy scripts, and it should be up on the S3 link
  8. To map this URL to a domain name, go to DNS provider (godaddy) and update the details

regd Back end

  1. Create a new IAM user (this is not necessary, but will make for a clean, new system)
  2. Change the cross_env_var to include the new github repo and new IAM user
  3. Change the variables inside group_vars - you will choose machine size here
  4. Now you can run the create_webserver script and the deploy_webserver script! YAAAY!
  5. Post this, run the script to spin up the server

Trouble shooting commands for python3 upgrade:

Clear space df -h sudo apt-get clean sudo apt clean cd /home/ubuntu/proto-2/server npm dedupe cd /usr/src ls -lt -- Delete older header like linux-aws-headers-4.4.0-1047 sudo apt purge linux-aws-headers-4.4.0-1079 sudo apt-get install linux-aws-headers-4.4.0-1095 sudo find / -name "linux-aws-*" sudo apt-get autoremove sudo apt autoremove sudo apt autoclean

Install pip3 sudo apt remove --purge python3-pip curl -O https://bootstrap.pypa.io/pip/3.5/get-pip.py sudo -E python3 get-pip.py