Move to a microservice architecture

[ifed3]

---

name: Upgrade WOAQ infrastructure
about: #122
title: Set up microservices
labels: infrastructure, api, web, dataviz
reviewers: @r-b-g-b, @theecrit, @nbilenko, @TangoYankee

---

Checklist

• Add description
• Reference open issue pull request addresses
• Pass linting check
  • complete on the local machine with make local.shell make quality
• Pass functional tests
  • complete on the local machine with make local.shell make test
• Request code review
  • Please allow 36 hours from opening a pull request before merging a pull request- even if it has already recieved an approving review.
• Address comments on code and resolve requested changes
• Merge own code

Description

Issue: #122

Thanks for your patience. This PR is finally ready.

• There are now five services in the WOAQ architecture: django rest api, react client, jupyter notebook for dataviz, postgis database, reverse proxy with automatically generated ssl certificates
• Services are no longer accessible by ports, instead they are defined by domains
  • Web interface is available on lvh.me
  • API is available on api.lvh.me
  • Data visualization notebook is available on dataviz.lvh.me
• README has been cleaned up to make it easier to get up and running. Project overview and historical info have been moved to the About.md section
• Python dependency manager has been changed from pipenv to pip-tools. It's simpler for working with Docker environments. Read more here
• Make commands that relate to docker have been removed due to refactored docker-compose files
  and multiple services instead of a single one
  • Run docker-compose up -d --build instead of make local.build + make local.up to build and run all services
  • Run docker-compose up -d --build service_name to run a specific service
  • Run docker-compose down instead of make local.down
  • Run docker-compose run service_name /bin/bash instead of make local.shell to enter the shell of a specific service
• One thing to pay attention to is the change in directory structure. All services have their own directory. Make sure to cd into the correct directory if you want to run commands for a specific service

p.s there are still some outstanding issues relating to devops/infrastructure that will need to be completed in the near future: writing tests for the client service, creating a Heroku container configuration file, setting up the Heroku deploy process, pushing container images to Docker hub to enable faster CI builds

[nbilenko]

Lots of awesome work here @ifed3. The infrastructure work looks great. The cleaner directory structure and separate Dockerfiles make it much easier to understand the codebase and to isolate changes related to different microservices.

However, there are changes that are out of scope of the issue, and this makes this PR quite large and more difficult to review. I would suggest retaining the work related to setting up microservices in this PR and separating out the other changes into different issues/PRs.

Some examples of changes I would suggest relocating:

• Python dependency management. I would be totally open to a change, but it's out of scope of this issue.
• Removal of make commands. Retaining make commands shouldn't impact anything about docker-compose commands since they are just aliases. It also seems unrelated to the issue.
• Major README cleanup. The cleanup looks great! But general cleanup for easier onboarding should be a separate issue. The changes in this PR should only be related to the microservice related changes.
• Adding a jupyter notebook. I would be concerned about running a jupyter notebook in production because of how computationally expensive it would be. Also, notebooks are great for interactive data analyses, but very bad for displaying pre-made visualizations. In any case, this change is out of scope of this issue.

Since I'm asking for a major PR change, I'll hold off on specific review of the infrastructure changes, but I'm looking forward to it - seems like a great way to move forward.

[theecrit]

I agree with @nbilenko here; we really want to make sure our issues contain discrete implementations so work can be tracked more easily and discussions can remain manageable. The issue breakdown provided feels appropriate to me, too.

Thanks,
J.

[ifed3]

Good catch @nbilenko with the dataviz service running in production. I've updated that.

I think it would be misguided to say that changes are out of scope of this refactor. They are interwoven in a manner that would require more work to separate them out in a new PR.

My aim was to build an architecture that can be run locally and on production without major configuration, so with microservices, three high level changes happened:

1. The only supported way to run the infrastructure is via docker, i.e you can't run it directly on your host machine
2. The only environment to run locally via docker is the dev/test environment. There will never be a need to run the prod or CI env locally
3. Subdirectories now exist thus each directory should only contain files relevant to its service

Switch from pipenv to pip-tools

pipenv by default provides a virtual env, serves as both a dependency manager and installer. This is ideal if you want to run a python app on your host machine but is problematic for docker usage.

pipenv uses different install commands based on the env: pipenv install --dev vs. pipenv install. To solve this would require creating a different Dockerfile to run the api service in prod simply due to one command change. Or to create a conditional bash script to check the env and run the appropriate command.

Once there is no need to run the infra on your host machine, it actually becomes both unnecessary and more cumbersome to maintain support for pipenv.

Makefile docker-compose aliases

Once the django app is moved to the api directory, it takes the Makefile with it since most of the commands are python/django related.

To keep the aliased docker commands would require creating a new Makefile within the root directory. The aliases were ideal when you could choose an env to run (local vs. production) but became redundant when you're limited to only run the local env

So you go from:

make %.up: ## Start the (local|production) Docker containers
    docker-compose -f docker-compose.yml -f docker-compose.$*.yml up -d

to:

make local.up ## Start the local Docker containers
    docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d

However by default, docker-compose searches for two config files (docker-compose.yml, docker-compose.override.yml) when running; once I renamed docker-compose.local.yml to docker-compose.override.yml, these commands all became equivalent!

make local.up
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d
docker-compose up -d

Thus all the aliases essentially became plain and easy to use docker-compose commands and getting familiar with docker-compose syntax will be far more important for one's coding chops than using arbitrary aliases especially for such basic commands.

So the aliases became counterproductive both from a code functionality and developer usage standpoint with a microservices architecture

README/contributing.md

These changes relate to the onboarding and use of the micoservices architecture. If I had left them as they were, half of it would be irrelevant and non-functional since it assumes just a django app rather than multiple apps.

Holoviews

This is the change I agree is extraneous and probably out of scope.

To your point, @nbilenko, Holoviews was built specifically to solve the problem of displaying pre-made visualizations. Read more here. It can output static displays in many formats such as png, jpeg, svg, html and dynamic displays as html + js. because it uses Bokeh as its backend provider which is described as follows:

Bokeh is an interactive visualization library that targets modern web browsers for presentation. Its goal is to provide elegant, concise construction of versatile graphics, and to extend this capability with high-performance interactivity over very large or streaming datasets

---

Hope that explains some of the changes. I think this conversation would be best be expounded upon and resolved in person. See you all tonight!

[nbilenko]

I think it would be misguided to say that changes are out of scope of this refactor. They are interwoven in a manner that would require more work to separate them out in a new PR.

The interweaving of the changes is exactly what my feedback is about - the interweaving complicates both the review and keeping track the history of these changes. The different changes I mentioned are hard to delineate and to keep track of while reading the code. Also, discussion for all changes has to be maintained in a single comment thread. E.g. a chat about pros and cons of dataviz software sounds great, but an infrastructure PR comment thread is not where decisions should be made about what software the team will be using for data visualization.

I realize it would take some work for you to separate out the changes, but to read and understand these interwoven changes would instead place a substantial amount of work on the rest of the team. It would also likely to not lead to effective review - best practices tend to recommend limiting code changes to a few hundred lines because reviewers can't reliably detect problems with longer change lists.

Let's discuss more tonight.

[ifed3]

hey all,

This is the new infra document

Here's a more thorough breakdown of the changes that happened within this PR. If you have any questions or any confusion, please let me know.

Docker

• Remove all files related to AWS, Terraform, Elastic Beanstalk
• Move django files to api directory
• Create react client service within web directory
• Replace nginx service and configs with fully managed nginx-proxy service
• Use nginx-proxy to map ports to domains
• Set up CORS configuration for nginx-proxy so api domain can be accessed by any other domain
• Use nginx-proxy-letsencrypt-companion to automagically set up SSL certificates for domains
• Rename docker-compose.local.yml to docker-compose.override.yml
• Rename docker-compose.travis.yml to docker-compose.ci.yml to align with environment
• Create docker-compose.prod.yml to set up production environment
• Remove docker-compose commands from Makefile due to django directory change and renaming of local compose override file
• Update dockerignore files to minimize unncessary files being copied to containers
• Drop pipenv to avoid api Dockerfile duplication for multiple environments
• Remove pipenv environment variables from api Dockerfile
• Remove logging configuration in api Dockerfile and entrypoint script in favor of PYTHONUNBUFFERED env variable and nginx-proxy taking care of gunicorn logs

CI (Travis)

• Replace wait-for-it script with dockerize library to avoid mantainance issues
• Await database connection with dockerize wait command to prevent CI build/test failure
• Setup secure environment variables to be used on Travis CI
• Simplify CI build and test scripts
• Update code coverage script

Wiki

• Clean README to set focus on awareness of available services and getting started quickly
• Move project details and history to About section
• Update contributing section to remove instructions on running on host machine (now only supported via docker-compose)
• Update contributing section to remove details about logging in to django admin (not needed for api usage)
• Update contributing section to provide a clear step-list for getting from issue to pull request
• Update contributing section to provide a one-liner for testing the api service without using aliases

Python

• Replace Pipfile/Pipfile.lock with requirements.in/requirements.txt
• Set python version with runtime.txt since Pipfile no longer exists
• Fix failing test from get_target_data method due to new pandas lib version being timezone aware
• Fix failing test from test_joiner and load_dustrak methods due to complex newline split script

[nbilenko]

Thank you for the detailed breakdown of the changes and documentation, Ife! I added inline comments about changes I had to make to get things running. Overall, looks great to me.

[r-b-g-b]

Very cool @ifed3 ! Thanks for this. I learned a lot from reviewing this -- extending docker-compose files, dockerize, pip-tools.

I did my best to understand the web service, SSL, letsencrypt, etc., but can't claim to be very knowledgeable about that, so I'll trust that you made it work. At the very least, I was able to get it up and running using the nice docs you provided.

I second @nbilenko 's point about preferring smaller PRs, so let's keep that in mind in the future. Approved!