crypto-stream-pi

Table of Contents

• Project Overview
• Requirements
• Project Structure
• Raspberry Pi Setup
  • Download Image
  • Configure SSH
  • Flash the Image to a microSD Card
  • Prepare the Raspberry Pi
  • Connect via SSH
• Build the Project
  • Cross Compilation
  • Host Build
• Running the Project
  • Running on a Real Raspberry Pi
    • Running Manually
    • Running Automatically on Boot
  • Running in QEMU
• Streaming Results
  • BTC-USDT Trades Price
  • BTC-USDT Trades Volume
  • BTC-USDT Volume Weighted Average Price (VWAP)
  • BTC-USDT Volume Moving Average
• Regenerating Plots

Project Overview

This project implements a real-time data processing system on a Raspberry Pi that monitors a set of major cryptocurrency instruments: BTC-USDT, ADA-USDT, ETH-USDT, DOGE-USDT, XRP-USDT, SOL-USDT, LTC-USDT, and BNB-USDT.

Trade data are fetched directly from the OKX exchange via a persistent WebSocket connection to the business stream endpoint: wss://ws.okx.com:8443/ws/v5/business (subscribing to the trades-all channel).
This feed was chosen because it provides every individual trade (unlike the public stream, which aggregates executions).

At its core, the system:

• Monitors live trades for the selected instruments with full granularity.
• Computes analytics every minute, including the 15-minute moving average (MA) of volume and Volume Weighted Average Price (VWAP).
• Calculates correlations across instruments based on recent VWAP values.
• Generates short-term price predictions for the next minute.
• Logs system diagnostics (CPU, memory, storage usage) to ensure robust operation on the target hardware.

Requirements

• A Raspberry Pi Board
• An Ethernet cable (for the first connection)
• CMake (for building the project)

Project Structure

• .github/: GitHub Actions workflows for CI/CD pipelines
• .vscode/: VS Code workspace configuration files
• ci/: Scripts for automated testing used in CI/CD pipelines
• docs/: Project documentation
• include/: Public header files
• lib/: Third-party lightweight libraries for JSON parsing (cJSON) and hash tables (uthash)
• scripts/: Utilities for configuration and cross-compilation
• src/: Main source code of the project

Raspberry Pi Setup

Download Image

Download the Raspberry Pi OS Lite from the official site for headless use and compatible with your Raspberry Pi model.

wget https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2025-05-13/2025-05-13-raspios-bookworm-arm64-lite.img.xz

Extract the image file

unxz raspios.img.xz

Configure SSH

Use the provided script to add a public ssh key to the image before flashing it to the SD card:

cd scripts
chmod +x ./rasp-config.sh 
./rasp-config.sh <image> <pubkey>

• Replace <image> with the image file
• Replace <pubkey> with your public ssh key.

⚠️ Attention: This script assumes the presence of a default pi user.
This is true for most fresh Raspberry Pi OS (Raspbian) images.
If the pi user does not exist, the script exits immediately.

Note: You will be prompted to enter your sudo password for mounting the image

Flash the Image to a microSD Card

Insert your microSD card and find its device name:

lsblk

Unmount all the partitions of the device (if any mounted)

sudo umount /dev/mmcblk1p*

Flash the image (replace /dev/mmcblk1 with your card device):

sudo dd bs=4M if=raspios.img of=/dev/mmcblk1 status=progress conv=fsync
sudo sync

⚠️ Caution: This command will completely overwrite the target device (/dev/mmcblk1) with the raspios.img image. Make absolutely sure you have selected the correct device, or you risk losing all data on it. Always double-check lsblk or fdisk -l before running dd.

Verify the image layout on the SD card. You should see the boot and root partition.

sudo fdisk -l /dev/mmcblk1

Pepare the Raspberry Pi

1. Insert the SD card into the Raspberry Pi

2. Connect the Raspberry to your router via an Ethernet cable

   In a fresh Raspbian image the WiFi is disabled by default by rfkill. Therefore, the first connection can only be achieved via Ethernet. You can configure connection via WiFi by running rasp-config on the Pi's terminal.

3. Power Up the Raspberry

   The Raspberry will automatically boot the OS stored inside the SD card.

Connect via SSH

Connect via ssh to the Raspberry Pi. On Linux, the Raspberry's IP is resolved by mDNS.

ssh pi@raspberrypi.local

Build the Project

Cross Compilation

The project has been cross compiled accross various architectures and OSes as show in the table below. For each OS and architecture, a proper cross-compilation toolchain should be selected. You can download the proper toolchain based on your target architecture and the image OS from here.

OS	Architecture	Compiler
Raspbian 9 (Stretch)	armv4	8.3.0
Raspbian 10 (Buster)	armv4	8.3.0
Raspbian 11 (Bullseye)	armv4	10.2.0
Raspbian 12 (Bookworm)	aarch64	14.2.0

To cross compile the project, run the following script:

cd scripts
chmod +x cross-compile.sh
./cross-compile.sh <image> <prefix> [--enable-wal]

Where

• image: The full path to the image file
• prefix: The full path to the cross compiler prefix. For example, if the downloaded compiler is in cross-gcc-8.3.0 the prefix would be something like full/path/to/cross-gcc-8.3.0/bin/arm-linux-gnueabihf-
• --enable-wal: Enables WAL mode with NORMAL synchronous setting for the trades SQLite databases. By default, the databases use DELETE mode with FULL synchronous setting.

⚠️ Attention: Make sure you have passed the trailing - to the prefix

The script does the following:

• Mounts the image
• Extracts the system directories that are necessary for the compilation i.e. (/usr, /lib) in tmp/sysroot/.
• Unmounts the image
• Fixes any broken symlinks to point inside tmp/sysroot/ and creates new ones used by the cross-compiler toolchain.
• Downloads the neccesary dependency libaries (SQLite, OpenSSL and libwebsockets) inside tmp/.
• Installs the static libaries and their headers to tmp/sysroot/
• Build the actual project (the binary is: build/cspi)

Note: The script will prompt you for the sudo password in order to mount the image

Note: During cross-compilation, the image file itself is not modified. The resulting binary is statically linked with all necessary libraries, so it contains all required code. You can simply upload the binary to the Raspberry Pi, there is no need to re-flash the entire image.

Host Build

You can also build the project to run on your host machine via CMake.

cmake -S . -B build && cmake --build build

You can enable WAL mode with the NORMAL synchronous setting at build time by passing
-DENABLE_WAL_NORMAL=ON to CMake.

⚠️ Attention: For host builds, you have make sure that the required libraries (OpenSSL, SQLite and libwebsockets) are installed on your computer. In cross-compilation, those dependencies are resolved automatically.

Running the Project

Running on a Real Raspberry Pi

Running Manually

After cross-compiling the project, you can upload the binary cspi to your Raspberry Pi (assuming SSH is configured):

scp <build/cspi> pi@raspberrypi.local:/home/pi/

Then, on the Raspberry Pi, set the executable permission and run it:

chmod +x /home/pi/cspi
/home/pi/cspi

Running Automatically on Boot

You can configure the binary to run aytomatically on boot by passing the binary to the rasp-config.sh script:

cd scripts
./rasp-config <image> <pubkey> <binary>

This injects the binary in /home/pi/crypto-stream-pi/ and sets up a systemd service named crypto-stream.service to start it automatically on boot. By default, the program is configured to run for 2 days.

You can check the status of the service by:

sudo systemctl status crypto-stream.service

Running in QEMU

We also provide a script for simulating a Raspberry Pi on Stretch, Buster, and Bullseye 32-bit environments. While originally intended for CI/CD automated testing, it can be used for local emulation as well.

cd ci
chmod +x qemu.sh
./qemu.sh <image> <binary>

• Replace <image> with your image file
• Replace <binary> with the compiled binary.

The script performs the following steps:

• Mounts the image and injects the binary file.
• Detects the Raspbian version from the image (supports: Stretch, Buster and Bullseye).
• Unmounts the image.
• Downloads the appropriate kernel version and PTB file from this repo.
• Creates a systemd service to launch the binary automatically on boot inside QEMU.
• Creates a systemd service to shut down the QEMU session after the binary finishes executing.
• Mounts back the image and tests the output files generated by the binary.
• Unmounts the image and exits.

⚠️ Note: Ensure that your image is not mounted elsewhere on the host to avoid read-only errors, and that you have sudo privileges to mount loop devices.

Streaming Results

The following results were obtained after running the program continuously for 2 days
on a Raspberry Pi 5 (4GB RAM, 64-bit) with Raspbian 12 (Bookworm) and a 64GB SanDisk SD Card.

A tarball containing the complete program output is available here

BTC-USDT Trades Price

BTC-USDT Trades Volume

BTC-USDT Volume Weighted Average Price (VWAP)

BTC-USDT Volume Moving Average

Plots for the other instruments can be found in the docs/plots/ directory of this repository.

Regenerating Plots

To regenerate all plots from the output directory tree, run:

cd docs
python3 plot_results.py <logs_dir> <output_dir>

• Replace <logs_dir> with the directory where you extracted the program output.
• Replace <output_dir> with the directory where the plots should be saved.