SkyViz

Stores data streams from ADSB receivers to generate insights about the world above us.

Note: This is a WIP, more visualizations soon to come!

skyviz.app »

Table of Contents

1. Project Overview
2. Getting Started
3. Development
   • Testing
4. Deployment
   • Staging
5. Database Migrations

Project Overview

The ADS-B Exchange network collects telemetry from plane transponders and publishes the current state of the airspace through their APIs. This project stores and transforms the data to make it available for historical analysis in a Streamlit web app.

Resources are containerized using Docker and hosted in Azure. CI/CD is enabled through Pulumi and Github Actions. Database versioning and migrations are handled by SQLAlchemy ORM and Alembic. Environment configuration is managed using Hydra.

$~$

(back to top)

$~$

Getting Started

1. Create a free subsciption to ADSBexchange APIs on RapidAPI
2. Clone the repo

   git clone https://github.com/fpvian/sky-viz.git

3. Rename .env.template to .env and fill with api key

   ADSB_EXCHANGE_API_KEY_DEV=<your key here>
   

4. Install Docker
5. Start containers

   docker compose up

6. Navigate to localhost:8501

(back to top)

$~$

Development:

1. Install Python
2. Create and activate a virtual environment

   python3 -m venv .venv --upgrade-deps
   source .venv/bin/activate

3. Install packages and dependencies

   pip install --editable .

4. Run modules

   python3 -m flights &
   streamlit run ./src/skyviz/Home.py

(back to top)

$~$

Testing:

Tests are run automatically when opening a PR and merging to main. Coverage reports are added to the PR as a comment.

1. Add ADSB_EXCHANGE_API_KEY_DEV to the Github repository secrets
2. Run all tests and generate coverage report

   pytest

(back to top)

$~$

Deployment:

After following these steps, deployments will be handled automatically when merging to main. Note that webapp_name in src/flights/config/groups/general.py and server_name in src/flights/config/groups/db.py must be globally unique in Azure.

1. Create Pulumi account and access token
2. Create Azure account and service principal with "Contributer" role
3. Create Docker Hub account and access token
4. Populate ARM, PULUMI, DOCKER, POSTGRES, and ADSB fields in .env file and GitHub repository secrets

   ARM_CLIENT_ID=<your azure service principal client id>
   ARM_CLIENT_SECRET=<your azure service principal client secret>
   ARM_TENANT_ID=<from `az account list`>
   ARM_SUBSCRIPTION_ID=<id from `az account list`>
   PULUMI_ACCESS_TOKEN=<your pulumi token>
   DOCKER_USER=<your docker username>
   DOCKER_TOKEN=<your docker token>
   POSTGRES_USERNAME=<make up a username>
   POSTGRES_PASSWORD=<randomly generate a password>
   ADSB_EXCHANGE_API_KEY_DEV=<your key here>
   ADSB_EXCHANGE_API_KEY_PROD=<your key here>
   

5. Install Pulumi

   curl -fsSL https://get.pulumi.com | sh

6. Run Pulumi

   cd deployment/
   export SKYVIZ_ENV=prod
   pulumi up

7. Update DNS records and issue certificates for domain
8. Install Pulumi app in GitHub repository to preview changes to prod resources as a PR comment

(back to top)

$~$

Staging:

Staging environment is automatically deployed when opening a pull request to main and automatically destroyed when merged. Follow these steps to manually manage staging environment.

1. Run Pulumi in a separate stack

   cd deployment/
   export SKYVIZ_ENV=staging
   pulumi up

2. Navigate to Staging Website
3. Clean up resources

   pulumi destroy

(back to top)

$~$

Database Migrations:

1. Navigate to folder with alembic.ini file

   cd src/flights/db/

2. Set environment variable to select database

   export SKYVIZ_ENV=test  # ephemeral sqlite database

3. Check for changes to the database model

   alembic check

4. Create a new version

   alembic revision --autogenerate -m "<name of commit>"

5. Test migration (migrations are handled automatically)

   alembic upgrade heads

6. Undo the last migration

   alembic downgrade -1

7. Restore database to its initial state

   alembic downgrade base

(back to top)