Mini MDS

Lightweight, open source, locally-hosted Modern Data Stack

• Extract & Load: dlt
• Data Quality: Great Expectations
• Storage: DuckDB
• Transformation: dbt
• Orchestration: Prefect
• Visualization: Dash

Installation

Clone repository and change directory:

git clone https://github.com/esadek/mini-mds.git
cd mini-mds

Install required packages:

pip install -r requirements.txt

Add dbt connection profile:

python scripts/add_profile.py

Usage

Extract, validate, load and transform data:

python prefect/elt.py

Visualize data:

python dash/app.py

Project Structure

mini-mds
├── .github/                # GitHub workflows
├── dash/                   # Dash application
├── dbt/                    # dbt project
├── duckdb/                 # DuckDB warehouse
├── prefect/                # Prefect workflows
├── scripts/                # Scripts
├── .gitignore              # Untracked files to ignore
├── LICENSE                 # MIT license
├── README.md               # Documentation
└── requirements.txt        # Python dependencies