The 311 Data Server is comprised of several different components deployed in containers to the public cloud. The API is intended to be modern, asynchronous, simple, performant, secure, open-source and well-supported.
The server code is set up as a monorepo with directories and containers corresponding to the following functions:
- RESTful Python API (/api)
- Report Server (/dash)
- Plotly Dash: for interactive, responsive reports
- Pandas: for pulling, aggregating, and calculating data in reports
- Data loading pipeline (/prefect)
- Prefect: for the nightly data ingestion tasks and cache clearing and priming
The last 2 projects are:
Everything runs on Python 3.7 and the Debian 10 "Buster" Linux distribution where applicable.
Data comes from the LA 311 system by way of the Socrata app using a Prefect workflow that is run nightly. Prefect is deployed in its own container and is run as a nightly Scheduled Task.
For more information about the data loading process look at the Prefect README.
Unit and integration testing is done locally and as a CI process run using GitHub Actions.
There are several testing tools used in the project:
- pytest: unit and integration tests
- locust: load tests
We currently have 90+% coverage on the API. To get code coverage reports run pytest --cov=code
Containers are deployed on merge to Docker Hub and new tasks are created in ECS. The result is a zero-downtime deployment. When a new version requires a database change a separate task must be created to run upgrades using Alembic.
The infrastructure is deployed to AWS using Terraform. API containers are hosted in ECS with data served from an RDS instance. For more information about the infrastrucure deployment process look at the Terraform README.