About this repo

A simple sample on how to use Google Composer and Kubernetes Pod Operator

airflow-kubernetes-pod-operator

A simple sample on how to use Airflow with KubernetesPodOperator base on Airflow on Kubernetes (Part 1): A Different Kind of Operator

Usefull links:

• Samples
• Source Code
• Pod Operator in Cloud Composer
• Airflow Operator

How to use this sample

• Prepare deploy

mkdir sample
find -name '*.py' -o -name '*.sql' | xargs cp -t sample/

• Download and prepare requirements (if need)

mkdir pkg
pip install --install-option="--install-lib=$PWD/pkg" -r requirements.txt
mv pkg/ sample/pkg

• Deploy dag

gcloud composer environments storage dags import \
                            --project <project> \
                            --environment <cluster> \
                            --location <location> \
                            --source sample

• Check your Airflow Web UI :)