Add documentation on testing BigQuery from ingestion-beam

.mermaid

@@ -0,0 +1,3 @@
+{
+    "themeCSS": ".label foreignObject { overflow: visible; }"
+}

.spelling

@@ -29,6 +29,7 @@ failsafe
 featureful
 filesystem
 GCP
+GCS
 GeoIP
 GKE
 GroupByKey
@@ -58,11 +59,13 @@ protobuf
 PubSub
 PubsubMessage
 Q4
+readme
 Redis
 Republisher
 runtime
 S3
 schemas
+SDK
 sharding
 SQLite
 stderr

bin/update-diagrams

@@ -12,5 +12,6 @@ for f in $(find . -name "*.mmd"); do
            --volume $PWD:/root/project \
            --workdir /root/project \
            $IMAGE \
+           -c .mermaid \
            -i ${f} -o ${f/.mmd/.svg}
 done

docs/diagrams/workflow.mmd

@@ -0,0 +1,27 @@
+graph TD
+
+subgraph dataops/sandbox/my-project
+    dataflow
+    bigquery
+    pubsub
+    subscription
+
+    pubsub --> |gcloud pubsub subscriptions| subscription
+    subscription --> dataflow
+    dataflow --> bigquery
+end
+
+subgraph mozilla-pipeline-schemas
+    mps[repository archive]
+end
+
+subgraph ingestion-beam
+    src[src/]
+    schemas[schemas.tar.gz]
+    bq-schemas[bq-schemas/]
+
+    src --> |mvn compile| dataflow
+    mps --> |download-schemas| schemas
+    schemas --> |generate-bq-schemas| bq-schemas
+    bq-schemas --> |update-bq-table| bigquery
+end

docs/ingestion_testing_workflow.md

@@ -0,0 +1,85 @@
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+
+
+- [Ingestion Testing Workflow](#ingestion-testing-workflow)
+  - [Setting up the GCS project](#setting-up-the-gcs-project)
+  - [Bootstrapping schemas from `mozilla-pipeline-schemas`](#bootstrapping-schemas-from-mozilla-pipeline-schemas)
+  - [Building the project](#building-the-project)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Ingestion Testing Workflow
+
+The ingestion-beam handles data flow of documents from the edge into various
+sinks. You may be interested in standing up a small testing instance to validate
+the integration of the various components.
+
+![diagrams/workflow.mmd](diagrams/workflow.svg)
+__Figure__: _An overview of the various components necessary to query BigQuery
+against data from a PubSub subscription._
+
+## Setting up the GCS project
+
+Read through [`whd/gcp-quickstart`](https://github.com/whd/gcp-quickstart) for details
+about the sandbox environment that is provided by data operations.
+
+* Install the [Google Cloud SDK](https://cloud.google.com/sdk/)
+* Navigate to the [Google Cloud Console](https://cloud.google.com/sdk/)
+* Create a new project under `firefox.gcp.mozilla.com/dataops/sandbox`
+    - `gcloud config set project <PROJECT>`
+* Create a PubSub subscription (see `gcp-quickstart/pubsub.sh`)
+* Create a GCS bucket
+    - `gsutil mb gs://<PROJECT>`
+* Enable the [Dataflow API](https://console.cloud.google.com/marketplace/details/google/dataflow.googleapis.com)
+* Create a service account and store the key locally
+
+
+## Bootstrapping schemas from `mozilla-pipeline-schemas`
+
+* Download the latest schemas from `mozilla-pipeline-schemas` using `bin/download-schemas`.
+    - This script may also inject testing resources into the resulting archive.
+    - A `schemas.tar.gz` will appear at the project root.
+* Generate BigQuery schemas using `bin/generate-bq-schemas`.
+    - Schemas will be written to `bq-schemas/`.
+    ```
+    bq-schemas/
+    ├── activity-stream.impression-stats.1.bigquery.json
+    ├── coverage.coverage.1.bigquery.json
+    ├── edge-validator.error-report.1.bigquery.json
+    ├── eng-workflow.bmobugs.1.bigquery.json
+    ....
+    ```
+* Update the BigQuery table in the current project using `bin/update-bq-table`.
+    - This may take several minutes. Read the script for usage information.
+* Verify that tables have been updated by viewing the BigQuery console.
+
+
+## Building the project
+
+Follow the instructions of the project readme. Here is a quick-reference for a running a job from a set of files in GCS.
+
+```bash
+export GOOGLE_APPLICATION_CREDENTIALS=keys.json
+PROJECT=$(gcloud config get-value project)
+BUCKET="gs://$PROJECT"
+
+path="$BUCKET/data/*.ndjson"
+mvn compile exec:java -Dexec.args="\
+    --runner=Dataflow \
+    --project=$PROJECT \
+    --autoscalingAlgorithm=NONE \
+    --workerMachineType=n1-standard-1 \
+    --numWorkers=1 \
+    --gcpTempLocation=$BUCKET/tmp \
+    --inputFileFormat=json \
+    --inputType=file \
+    --input=$path\
+    --outputType=bigquery \
+    --output=$PROJECT:\${document_namespace}.\${document_type}_v\${document_version} \
+    --bqWriteMethod=file_loads \
+    --tempLocation=$BUCKET/temp/bq-loads \
+    --errorOutputType=file \
+    --errorOutput=$BUCKET/error/ \
+"
+```

ingestion-beam/README.md

@@ -562,6 +562,9 @@ use the `bin/mvn` executable to run maven in docker:
 ./bin/mvn clean test
 ```
 
+To run the project in a sandbox against production data, see this document on
+![configuring an integration testing workflow](../docs/ingestion_testing_workflow.md).
+
 # License
 
 This Source Code Form is subject to the terms of the Mozilla Public