File tree Expand file tree Collapse file tree 2 files changed +24
-0
lines changed Expand file tree Collapse file tree 2 files changed +24
-0
lines changed Original file line number Diff line number Diff line change 2
2
3
3
![ DAG] ( ./doc/starcraft-duckdb.png )
4
4
5
+ This project is not only a source for knowledge sharing but also a celebration of my love for gaming and the
6
+ endless possibilities that data holds. It is a fusion of our two great passions: gaming and Data Engineering. I grew up
7
+ playing StarCraft: Brood War as well as StarCraft II a lot. I never made it to the grandmaster ladder, but I enjoyed every
8
+ match experiencing the adrenaline rush of commanding armies, outmaneuvering opponents, and claiming victory
9
+ (from time to time at least).
10
+
11
+ Just as I fine-tuned my build orders and adapt to enemy tactics in StarCraft, I am now optimizing data pipelines,
12
+ analyze trends, and visualize insights as a Data Engineer. In this project I would like to share knowledge about three
13
+ useful technologies of today's modern data stacks, which are namely:
14
+
15
+ * ⏱️ ** [ Apache Airflow] ( https://airflow.apache.org/ ) ** : Platform for orchestrating and scheduling complex workflows.
16
+ * 🦆 ** [ DuckDB] ( https://duckdb.org/ ) ** : Lightweight and versatile analytical database.
17
+ * 🚀 ** [ Streamlit] ( https://streamlit.io/ ) ** : User-friendly framework for building interactive web applications.
18
+
19
+ This project is basically a StarCraft II data pipeline, where data is fetched from the StarCraft II API and stored in
20
+ DuckDB, orchestrated via Airflow. There is also a Streamlit app to visualize how the current grandmaster ladder
21
+ in StarCraft II looks like (spoiler: you will not find me in it).
22
+
23
+ In ` playground.py ` you find more examples how to utilize DuckDB and Pandas to analyze the data.
24
+
25
+ You can find more in my article: https://vojay.de/2024/03/14/starcraft-data-pipeline/
26
+
27
+ This is our final result:
28
+
5
29
## Airflow DAG
6
30
7
31
![ DAG] ( ./doc/dag.png )
You can’t perform that action at this time.
0 commit comments