Add quick start for Airflow on Docker #13660

mik-laj · 2021-01-14T00:55:47Z

Hello,

Adds a docker-compose file that contains a simple sample configuration that allows you to quickly run Airflow in a Docker environment. This is a part of #8605 (comment), but much simpler because .... novice users don't care at all about their environment, but need to get all components up and running quickly. In the next steps, we can work on adding wizards that will allow you to generate your own docker-compose file, but I think it will require some dependent documentation and this is not needed in this section of the documentation yet.

Part of: #8605

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

github-actions · 2021-01-14T01:52:50Z

The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest master or amend the last commit of the PR, and push it with --force-with-lease.

github-actions · 2021-01-14T02:13:35Z

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.

docs/apache-airflow/start/docker-compose.yaml

potiuk

This is great but the docker compose does not work on Linux and it requires very recent version of docker-compose which is not explained nor checked (and error messages are difficult to understand in case of older versions used). Unfortunately the most popular ( and our main "target" Linux distribution does not use that version of docker-compose so we need to find a way to make the users at least be aware of the version requirement and direct them download page..

feluelle

I am wondering if we should use a .env file to populate some docker-compose values (which is passed to docker-compose automatically if in same dir). For example I did it like this: https://github.com/feluelle/finance-data-builder/blob/main/docker-compose.yml#L42-L46

And then I have a simple .env file like the following:

DBT_POSTGRES_HOST=fdb_dbt_db
DBT_POSTGRES_USER=dbt
DBT_POSTGRES_PASSWORD=dbt
DBT_POSTGRES_DB=dbt
DBT_POSTGRES_PORT=5432

AIRFLOW_POSTGRES_HOST=fdb_airflow_db
AIRFLOW_POSTGRES_USER=airflow
AIRFLOW_POSTGRES_PASSWORD=airflow
AIRFLOW_POSTGRES_DB=airflow
AIRFLOW_POSTGRES_PORT=5432

AIRFLOW_USER=airflow
AIRFLOW_PASSWORD=airflow

With this it is quite easy to change the access to the container and the airflow user.

docs/apache-airflow/start/docker-compose.yaml

docs/apache-airflow/start/docker.rst

mik-laj · 2021-01-18T11:39:15Z

With this it is quite easy to change the access to the container and the airflow user.

I would like this example to be simple and only use one file. This is a quick start for beginners, so we shouldn't use too complex syntax and require many steps. I will try to think of something to be able to introduce basic parameterization without adding extra steps to the instruction.

potiuk · 2021-01-18T12:10:06Z

I would like this example to be simple and only use one file. This is a quick start for beginners, so we shouldn't use too complex syntax and require many steps. I will try to think of something to be able to introduce basic parameterization without adding extra steps to the instruction.

Agree with @mik-laj -> if we have a single file to use, this is soooo much easier to share and use via gists and other similar mechanisms, or simple links to file in the repo. I wish it was easy to do in our Dockerfiles, but this is rather complex there and you usually needs all the "surrounding" stuff like scripts, empty dirs etc.

mik-laj · 2021-01-22T19:11:28Z

TODO: I would like to add optional sections on how to use CLI with wrappers.
Similar to:
https://github.com/KlubJagiellonski/pola-backend/blob/master/scripts/aws-cli.sh
https://github.com/KlubJagiellonski/pola-backend/blob/master/scripts/aws-cli-dev.sh
https://github.com/mik-laj/presto-hive-kerberos-docker/blob/master/psql.sh
https://github.com/mik-laj/presto-hive-kerberos-docker/blob/master/mc.sh
https://github.com/mik-laj/presto-hive-kerberos-docker/blob/master/hive.sh

This in some cases will allow us to forget that everything is running in Docker.

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>

mik-laj · 2021-01-24T08:44:41Z

I am wondering if we should use a .env file to populate some docker-compose values (which is passed to docker-compose automatically if in same dir).

I added a few parameters to the docker-compose file, but I was very careful not to complicate this file.
I only added these parameters in a few cases:

If it was necessary to ensure compatibility with Linux, (AIRFLOW_UID, AIRFLOW_GID),
if I thought it could be a really frequently changed option (AIRFLOW_IMAGE_NAME),
if it is a good example of using parameterization (_AIRFLOW_WWW_USER_USERNAME, _AIRFLOW_WWW_USER_PASSWORD)

Each variable has a default value defined so the .env file is optional. I also didn't define variables for internal details, e.g. database passwords, because Docker provides network isolation.

potiuk · 2021-01-24T09:49:38Z

Looks good but I added a few fixups:

Missing .gitignore entries for logs/plugins/env/dags
The DOCKER_COMPOSE_ARGS are not needed IMHO . The docker compose run will allocate terminal as needed and you can disable it with -T flag. Also interactive mode works out-of-the-box with docker-compose-run. removed them
I removed airflow from docker-compose run in airflow.sh. It is not neeed, it does not work with airfllow 2.0 image and it does not allow to run the two useful python and bash commands
changed ${*} into "${@}" (that's the proper way of passing parameters containing spaces potentially)
I added examples with "python/bash" commands.

potiuk · 2021-01-24T09:50:01Z

Feel fre to modify them if you feel it can be done better.

potiuk · 2021-01-24T09:50:32Z

And it works now on linux nicely :)

github-actions · 2021-01-24T09:51:33Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.

docs/apache-airflow/start/docker-compose.yaml

docs/apache-airflow/start/docker.rst

turbaszek · 2021-01-24T10:21:36Z

docs/apache-airflow/concepts.rst

+  in reality it is NOT a separate process, but run within the Scheduler.
+
+* The **Worker(s)** are separate processes which also interact with the other components of the Airflow architecture and
+  the metadata repository.


Are workers required in case of every executor?

We support only celery executor in this guide.

I may have missed it but I could not found where we say it. In fact we say something totally different than CeleryExecutor:

Primarily intended for development use, the basic Airflow architecture with the Local and Sequential executors is an excellent starting point for understanding the architecture of Apache Airflow.

Ahh. And that's not my text. I only moved this paragraph from another place, but I will check it more carefully.

At the very beginning of this section it says this is the architecture for LocalExecutor and SeqentialExecutor, and both of these executors use a separate process to run the DAG code, so everything is correct.

Primarily intended for development use, the basic Airflow architecture with the Local and Sequential executors is an
excellent starting point for understanding the architecture of Apache Airflow.

feluelle · 2021-01-24T12:05:19Z

docs/apache-airflow/start/docker.rst

+
+    airflow-init_1       | Upgrades done
+    airflow-init_1       | Admin user airflow created
+    airflow-init_1       | 2.1.0


Is this the version? Why 2.1.0?

Further down we mention the version again for example apache/airflow:2.0.0. This should be the same, or?

It should be Airflow 2.0.1, but it is not unreleased yet. At this point, I used this version because I was running the image from the master branch where we have version 2.1.0.dev, then deleted the suffix ".dev".

airflow/setup.py

Line 41 in f473ca7

version = '2.1.0.dev0'

I am concerned that none of the released images works, as we have not yet voted on any release that includes the latest changes.
#13728
I will try to unify it so that there is one version used in both places.

2.0.0 works with my fixup but without creating admin user

I added extra checks for docs build: 341c781

docs/apache-airflow/start/docker.rst

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> (cherry picked from commit ffb472c)

mik-laj requested review from potiuk and kaxil January 14, 2021 00:55

boring-cyborg bot added area:dev-tools kind:documentation labels Jan 14, 2021

kaxil approved these changes Jan 14, 2021

View reviewed changes

github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Jan 14, 2021

potiuk reviewed Jan 14, 2021

View reviewed changes

docs/apache-airflow/start/docker-compose.yaml Show resolved Hide resolved

potiuk reviewed Jan 14, 2021

View reviewed changes

docs/apache-airflow/start/docker-compose.yaml Show resolved Hide resolved

potiuk reviewed Jan 14, 2021

View reviewed changes

docs/apache-airflow/start/docker-compose.yaml Show resolved Hide resolved

potiuk requested changes Jan 14, 2021

View reviewed changes

github-actions bot removed the okay to merge It's ok to merge this PR as it does not require more tests label Jan 14, 2021

feluelle reviewed Jan 18, 2021

View reviewed changes

docs/apache-airflow/start/docker-compose.yaml Outdated Show resolved Hide resolved

docs/apache-airflow/start/docker.rst Outdated Show resolved Hide resolved

mik-laj mentioned this pull request Jan 22, 2021

Setting up Airflow for local development is hard #13838

Closed

Kamil Breguła and others added 6 commits January 24, 2021 05:48

Add quick start for Airflow on Docker

e70b6ce

fixup! Add quick start for Airflow on Docker

0a326ff

fixup! fixup! Add quick start for Airflow on Docker

9c676bb

fixup! fixup! fixup! Add quick start for Airflow on Docker

b28b18c

Update docs/apache-airflow/start/docker-compose.yaml

16e12f8

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>

Rewrite db initialization and add section about CLI and volumes

664614d

mik-laj force-pushed the quick-start-docker branch from 2968044 to 664614d Compare January 24, 2021 07:35

Improve compatiblity with Linux

d1fdb57

mik-laj requested a review from potiuk January 24, 2021 08:46

fixup! Improve compatiblity with Linux

fe736be

mik-laj force-pushed the quick-start-docker branch from ee8aba0 to fe736be Compare January 24, 2021 09:40

potiuk added 4 commits January 24, 2021 10:44

fixup! Improve compatiblity with Linux

b53e5f7

fixup! fixup! Improve compatiblity with Linux

8d47216

fixup! fixup! fixup! Improve compatiblity with Linux

fdd3b42

fixup! fixup! fixup! fixup! Improve compatiblity with Linux

54d5740

potiuk approved these changes Jan 24, 2021

View reviewed changes

github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Jan 24, 2021

turbaszek reviewed Jan 24, 2021

View reviewed changes

docs/apache-airflow/start/docker-compose.yaml Show resolved Hide resolved

turbaszek reviewed Jan 24, 2021

View reviewed changes

docs/apache-airflow/start/docker.rst Show resolved Hide resolved

turbaszek reviewed Jan 24, 2021

View reviewed changes

fixup! fixup! fixup! fixup! fixup! Improve compatiblity with Linux

15189d9

feluelle reviewed Jan 24, 2021

View reviewed changes

Kamil Breguła added 2 commits January 24, 2021 16:24

Add checks for airflow version and image tag

341c781

fixup! Add checks for airflow version and image tag

de9a9e9

mik-laj commented Jan 24, 2021

View reviewed changes

docs/apache-airflow/start/docker.rst Outdated Show resolved Hide resolved

Update docs/apache-airflow/start/docker.rst

a3f78c6

kaxil reviewed Jan 25, 2021

View reviewed changes

docs/apache-airflow/start/docker.rst Outdated Show resolved Hide resolved

Update docs/apache-airflow/start/docker.rst

e79799f

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

mik-laj merged commit ffb472c into apache:master Jan 26, 2021

mik-laj deleted the quick-start-docker branch January 26, 2021 12:45

kaxil added this to the Airflow 2.0.1 milestone Jan 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add quick start for Airflow on Docker #13660

Add quick start for Airflow on Docker #13660

mik-laj commented Jan 14, 2021

github-actions bot commented Jan 14, 2021

github-actions bot commented Jan 14, 2021

potiuk left a comment

feluelle left a comment

mik-laj commented Jan 18, 2021 •

edited

Loading

potiuk commented Jan 18, 2021

mik-laj commented Jan 22, 2021 •

edited

Loading

mik-laj commented Jan 24, 2021

potiuk commented Jan 24, 2021 •

edited

Loading

potiuk commented Jan 24, 2021

potiuk commented Jan 24, 2021

github-actions bot commented Jan 24, 2021

turbaszek Jan 24, 2021

mik-laj Jan 24, 2021

turbaszek Jan 24, 2021

mik-laj Jan 24, 2021

mik-laj Jan 24, 2021

feluelle Jan 24, 2021

mik-laj Jan 24, 2021

potiuk Jan 24, 2021 •

edited

Loading

mik-laj Jan 24, 2021

Add quick start for Airflow on Docker #13660

Add quick start for Airflow on Docker #13660

Conversation

mik-laj commented Jan 14, 2021

github-actions bot commented Jan 14, 2021

github-actions bot commented Jan 14, 2021

potiuk left a comment

Choose a reason for hiding this comment

feluelle left a comment

Choose a reason for hiding this comment

mik-laj commented Jan 18, 2021 • edited Loading

potiuk commented Jan 18, 2021

mik-laj commented Jan 22, 2021 • edited Loading

mik-laj commented Jan 24, 2021

potiuk commented Jan 24, 2021 • edited Loading

potiuk commented Jan 24, 2021

potiuk commented Jan 24, 2021

github-actions bot commented Jan 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

potiuk Jan 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mik-laj commented Jan 18, 2021 •

edited

Loading

mik-laj commented Jan 22, 2021 •

edited

Loading

potiuk commented Jan 24, 2021 •

edited

Loading

potiuk Jan 24, 2021 •

edited

Loading