Skip to content

Learning resources for Airflow Tutorial article.

Notifications You must be signed in to change notification settings

dmlogv/airflow-tutorial

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

57 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Airflow Tutorial

Learning resources for Airflow Tutorial article.

Contents

  • docker-compose.yml — An example of Airflow cluster with Celery executor. Contains:
    • Apache Airflow
    • PostgreSQL (Airflow metadata)
    • Redis (Task broker)
    • Celery workers
    • Flower (Celery monitoring)
  • docker-compose.db.yml — Additional database servers and sample data fill up:
    • SQL Server x3 (source database servers)
    • Vertica (target database)
    • mssql_init (initialize source data)
  • dags/ — Sample DAGs and common libraries.

Start and stop

To spin up Airflow cluster only (without databases), use:

$ docker-compose up --scale worker=3

To run all described images and create sample databases, execute:

$ docker-compose -f docker-compose.yml -f docker-compose.db.yml up --scale worker=3

To break down containers press Ctrl+C or Command+C and the following command:

$ docker-compose down

or

$ docker-compose -f docker-compose.yml -f docker-compose.db.yml down

Usage

Containers exposes a couple of WebUI's: