Beam to BigQuery

This repo contains sample code for testing different BigQuery persistence methods in Apache Beam pipelines

Environment setup

export PROJECT_ID=<project_id>
export GCP_REGION=<region>
export BIGQUERY_REGION=us
source setup-env.sh

./start-event-generation.sh <events-per-second>

cd pipeline
./run-dataflow-<method>.sh <params>

For Storage Write API:

./run-dataflow-storage-streaming.sh <number-of-streams> <triggering-frequency-in-seconds>

Name		Name	Last commit message	Last commit date
Latest commit History 41 Commits
data-generator		data-generator
direct		direct
pipeline-schema-change		pipeline-schema-change
pipeline		pipeline
raw-api-ingestion		raw-api-ingestion
spanner-change-data-capture		spanner-change-data-capture
terraform		terraform
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
setup-env.sh		setup-env.sh
start-data-capture.sh		start-data-capture.sh
start-event-generation.sh		start-event-generation.sh