bastiaanb / twitter-analysis-on-gcp Public

Notifications You must be signed in to change notification settings
Fork 0
Star 2

PoC: setup of BigQuery and Dataflow on Google Cloud Platform for ingesting and analyzing tweets.

2 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 33 Commits
beam-jobs		beam-jobs
ingest-twitter		ingest-twitter
sample-data		sample-data
terraform		terraform
README.md		README.md
requirements.txt		requirements.txt

Repository files navigation

Twitter Analysis on Google Cloud Platform

Proof of Concept for ingestion of twitter streams into Google BigQuery via Google Dataflow

Project layout:

beam-jobs/ - Python Beam jobs for loading tweests into BigQuery and for generating a list of trending topics.
ingest-twitter/ - scripts to ingest Tweet streams
sample-data/ - some already collected tweets
terraform/ - Infrastructure setup

Getting started

Setup infra per terraform/ folder.
Get twitter stream data with scripts in ingest-twitter/ folder or use supplied sample data.
Copy twitter data to created Google Cloud Storage bucket.
Install python dependencies pip install -r requirements.txt
In beam-jobs/`` adjust parameters in run-import-on-gcp.sh` and run.

About

PoC: setup of BigQuery and Dataflow on Google Cloud Platform for ingesting and analyzing tweets.

Report repository

Releases

No releases published

Packages

No packages published

Languages