The ingestion process fetches a subset of the TfL data through Ably online source as part of a proof of concept data ingestion use case. It runs as 2 App Engine apps, one of which handles both fetching the data and forwarding it to Pub/Sub (Ingest Ably TFL), the other reads it from Pub/Sub and inserts it into BigQuery (this app).
The app configuration resides in a app.yaml template called app.yaml.tmpl. The reason for the template is to allow Cloud Build to inject environment variables into the configuration file if needed.
PROJECT: $ANALYTICS_PROJECT # replace in cloud build step
TOPIC: ably-tfl-tube
SUBSCRIPTION: ably-tfl-tube
DATASET: ably_tfl
TABLE: tube
The $ANALYTICS_PROJECT environment variable is a pipeline substitution in the pipeline trigger. It is injected as part of a Cloud Build step:
sed -e "s/\$$ANALYTICS_PROJECT/$_ANALYTICS_PROJECT/g" app.yaml.tmpl > app.yaml
It is passed through from the Platform Bootstrap process, which is where it is originally configured.
The pipeline is automatically triggered when code is pushed. It can also be triggered manually via the console.