Skip to content

Commit 117b585

Browse files
committed
Update data and extend README
1 parent b83c1a3 commit 117b585

File tree

2 files changed

+24
-0
lines changed

2 files changed

+24
-0
lines changed

README.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,30 @@
22

33
![DAG](./doc/starcraft-duckdb.png)
44

5+
This project is not only a source for knowledge sharing but also a celebration of my love for gaming and the
6+
endless possibilities that data holds. It is a fusion of our two great passions: gaming and Data Engineering. I grew up
7+
playing StarCraft: Brood War as well as StarCraft II a lot. I never made it to the grandmaster ladder, but I enjoyed every
8+
match experiencing the adrenaline rush of commanding armies, outmaneuvering opponents, and claiming victory
9+
(from time to time at least).
10+
11+
Just as I fine-tuned my build orders and adapt to enemy tactics in StarCraft, I am now optimizing data pipelines,
12+
analyze trends, and visualize insights as a Data Engineer. In this project I would like to share knowledge about three
13+
useful technologies of today's modern data stacks, which are namely:
14+
15+
* ⏱️ **[Apache Airflow](https://airflow.apache.org/)**: Platform for orchestrating and scheduling complex workflows.
16+
* 🦆 **[DuckDB](https://duckdb.org/)**: Lightweight and versatile analytical database.
17+
* 🚀 **[Streamlit](https://streamlit.io/)**: User-friendly framework for building interactive web applications.
18+
19+
This project is basically a StarCraft II data pipeline, where data is fetched from the StarCraft II API and stored in
20+
DuckDB, orchestrated via Airflow. There is also a Streamlit app to visualize how the current grandmaster ladder
21+
in StarCraft II looks like (spoiler: you will not find me in it).
22+
23+
In `playground.py` you find more examples how to utilize DuckDB and Pandas to analyze the data.
24+
25+
You can find more in my article: https://vojay.de/2024/03/14/starcraft-data-pipeline/
26+
27+
This is our final result:
28+
529
## Airflow DAG
630

731
![DAG](./doc/dag.png)

my_duckdb.db

268 KB
Binary file not shown.

0 commit comments

Comments
 (0)