The last year has been like a roller coaster for the cryptocurrency market. At the end of 2017, the value of bitcoin (BTC) almost reached $20,000 USD, only to fall below $4,000 USD a few months later. What if there is a pattern in the high volatility of the cryptocurrencies market? If so, can we learn from it and get an edge on future trends? Is there a way to observe all exchanges in real time and visualize it on a single chart?
In this tutorial we will graph the trades, volume and time delta from trade execution until it reaches our system (an indicator of how close to real time we can get the data).
Consider reading the Medium article
Terraform - get this up and running in less than 5 minutes
This tutorial uses billable components of GCP, including:
- Cloud Dataflow
- Compute Engine
- Cloud Storage
- Cloud Bigtable
We recommend to clean up the project after finishing this tutorial to avoid costs. Use the Pricing Calculator to generate a cost estimate based on your projected usage.
- Log into the console, and activate a cloud console session
- Create a new VM
gcloud beta compute instances create crypto-driver \
--zone=us-central1-a \
--machine-type=n1-standard-1 \
--service-account=$(gcloud iam service-accounts list --format='value(email)' --filter="compute") \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--image=debian-9-stretch-v20181210 \
--image-project=debian-cloud \
--boot-disk-size=20GB \
--boot-disk-device-name=crypto-driver
- SSH into that VM
gcloud compute ssh --zone=us-central1-a crypto-driver
- Installing necessary tools like java, git, maven, pip, python and Cloud Bigtable command line tool cbt using the following command:
sudo -s
apt-get update -y
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python3 get-pip.py
sudo pip3 install virtualenv
virtualenv -p python3 venv
source venv/bin/activate
sudo apt -y --allow-downgrades install openjdk-8-jdk git maven google-cloud-sdk=271.0.0-0 google-cloud-sdk-cbt=271.0.0-0
export PROJECT=$(gcloud info --format='value(config.project)')
export ZONE=$(curl "http://metadata.google.internal/computeMetadata/v1/instance/zone" -H "Metadata-Flavor: Google"|cut -d/ -f4)
gcloud services enable bigtable.googleapis.com \
bigtableadmin.googleapis.com \
dataflow.googleapis.com
gcloud bigtable instances create cryptorealtime \
--cluster=cryptorealtime-c1 \
--cluster-zone=${ZONE} \
--display-name=cryptorealtime \
--cluster-storage-type=HDD \
--instance-type=DEVELOPMENT
cbt -instance=cryptorealtime createtable cryptorealtime families=market
gsutil mb -p ${PROJECT} gs://realtimecrypto-${PROJECT}
gcloud compute firewall-rules create crypto-dashboard --action=ALLOW --rules=tcp:5000 --source-ranges=0.0.0.0/0 --target-tags=crypto-console --description="Open port 5000 for crypto visualization tutorial"
gcloud compute instances add-tags crypto-driver --tags="crypto-console" --zone=${ZONE}
git clone https://github.com/GoogleCloudPlatform/professional-services
cd professional-services/examples/cryptorealtime
mvn clean install
./run.sh ${PROJECT} \
cryptorealtime gs://realtimecrypto-${PROJECT}/temp \
cryptorealtime market
cd frontend/
pip install -r requirements.txt
python app.py ${PROJECT} cryptorealtime cryptorealtime market
Find your external IP in Compute console instance and open it in your browser with port 5000 at the end e.g. http://external-ip:5000/stream
You should be able to see the visualization of aggregated BTC/USD pair on several exchanges (without predictor part)
- To save on cost we can clean up the pipeline by running the following command
gcloud dataflow jobs cancel \
$(gcloud dataflow jobs list \
--format='value(id)' \
--filter="name:runthepipeline*")
- Empty and Delete the bucket:
gsutil -m rm -r gs://realtimecrypto-${PROJECT}/*
gsutil rb gs://realtimecrypto-${PROJECT}
- Delete the Cloud Bigtable instance:
gcloud bigtable instances delete cryptorealtime
- Exit the VM and delete it.
gcloud compute instances delete crypto-driver --delete-disks
-
View the status of your Dataflow job in the Cloud Dataflow console
-
After a few minutes, from the shell,
cbt -instance=<bigtable-instance-name> read <bigtable-table-name>
Should return many rows of crypto trades data that the frontend project will read for it's dashboard.