Skip to content

abdulmateen59/etl_airflow_spark

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

End-to-End Pipeline using Apache Airflow, Spark and Docker

Getting Started:

$ docker-compose up --build
    Airflow: http://localhost:8282
    Spark Master: http://localhost:8181

Prerequisite

spark_default must be specified before running DAG, it can be configured via Web UI Goto: Admin -> Connections

         Host: spark://spark
         Port: 7077

Architecture

Archtitecture

About

Airflow data orchestration pipeline

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published