AskOpenData has a React frontend and a Python backend. There is a finite set of questions which the user can view. Given a question ID, the backend provides preprocessed data for the corresponding plot. The frontend lays out and draws the final plot to match the user's screen size. Question search is typically handled by the backend (see below).
The frontend is a React application written in TypeScript. It uses Victory for plots, d3 for maps of Zürich, and emotion for styling.The source code is in N3XT191/open-data-frontend (this repository).
The frontend is a very typical React app made with create-react-app. To get a local development environment:
- Install Node.js (v14 LTS) and Yarn (classic)
- Clone this repo
- Run
yarnat the repository root (the folder withyarn.lock) to install dependencies from NPM - Run
yarn startto start a local development server for the frontend
Note that you will also need to start the backend for the app to work (see below).
The next time you want to use the development environment, it's enough to run yarn start.
All files must be formatted with Prettier, otherwise they will be rejected by CI. The easiest way to do that is on save using an editor plugin. You can also run yarn format to format all files.
The development build assumes that the backend is at http://127.0.0.1:8000. When building for production you will need to adjust that URL. This can be done using an environment variable: env REACT_APP_API_URL=https://ask-open-data.ch/api yarn build.
The backend is a Python application which uses FastAPI and pandas to serve preprocessed datasets. Preprocessing is done to keep startup and response times fast. The source code is in tehwalris/open-data-backend.
First set up Python and install dependencies. The steps here are for pip and a virtual environment, but you don't necessarily have to use those.
- Install Python (v3.9) and pip
- Clone the backend repo
- Create a virtual environment:
python3 -m venv .venv - Activate the virtual environment:
source .venv/bin/activate - Install dependencies:
pip install -r requirements.txt
Before you start the server, you need to download and preprocess the datasets.
- Download raw datasets:
./download-data.sh - Preprocess datasets:
python -m src.preprocess
The development server will run on port 8000 by default, which is what the frontend expects.
- Start a development server:
uvicorn main:app --reload
The next time you want to use the development environment, it's enough to run source .venv/bin/activate and uvicorn main:app --reload.
This is the most typical way to deploy, but it's not what we actually do for ask-open-data.ch. For this kind of deployment you would serve the backend with uvicorn behind a frontend proxy like nginx. The frontend would be built with yarn build and served with the same nginx. The disadvantage of this setup is that you need to run the Python server, which requires something like a VM to host it on.
This is the deployment strategy that we use for ask-open-data.ch. It is automated end-to-end in tehwalris/open-data-static. Since we have a finite number of questions, we can pre-request every URL from the API and save the responses. This is done by src/static.py in the backend. The result of that script is a folder which can be served using a static HTTP server, but to the frontend looks the same as the dynamic Python API server.
The final build and deployment is fully automated in GitHub Actions. The runs of this CI system are publicly visible (example). All the code is reproducibly built directly from source. The full datasets are downloaded directly from https://data.stadt-zuerich.ch and all preprocessing is run from scratch according to the latest code. The API responses are statically saved as described above. This whole bundle is then pushed to Vercel (a deployment platform and CDN for mainly static content) where it can be accessed through https://ask-open-data.ch.