Skip to content

Commit

Permalink
Get rid of Airflow 1.10 in Breeze (#15712)
Browse files Browse the repository at this point in the history 
Gets rid of Airflow 1.10 in Breeze and script/configuration.

We were still using occasionally the master version of Breeze to
run 1.10 version of Airflow, but this madness should end now when
we are approaching 2.1 release. All changes in Breeze were so far
ported to 1.10 but this is about the time to finish it.

(cherry picked from commit 57ad4af)
  • Loading branch information
@potiuk
potiuk committed May 8, 2021
1 parent af06c9c commit c4476d2
Show file tree
Hide file tree
Showing 23 changed files with 80 additions and 344 deletions.
112 changes: 17 additions & 95 deletions BREEZE.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -541,7 +541,7 @@ For all development tasks, unit tests, integration tests, and static code checks
**CI image** maintained on the DockerHub in the ``apache/airflow`` repository.
This Docker image contains a lot of test-related packages (size of ~1GB).
Its tag follows the pattern of ``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>-ci``
(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v1-10-test-python3.6-ci``).
(for example, ``apache/airflow:master-python3.6-ci`` or ``apache/airflow:v2-0-test-python3.6-ci``).
The image is built using the `<Dockerfile.ci>`_ Dockerfile.

The CI image is built automatically as needed, however it can be rebuilt manually with
Expand Down Expand Up @@ -638,7 +638,7 @@ The **Production image** is also maintained on the DockerHub in the
``apache/airflow`` repository. This Docker image (and Dockerfile) contains size-optimised Airflow
installation with selected extras and dependencies. Its tag follows the pattern of
``<BRANCH>-python<PYTHON_MAJOR_MINOR_VERSION>`` (for example, ``apache/airflow:master-python3.6``
or ``apache/airflow:v1-10-test-python3.6``).
or ``apache/airflow:v2-0-test-python3.6``).

However in many cases you want to add your own custom version of the image - with added apt dependencies,
python dependencies, additional Airflow extras. Breeze's ``build-image`` command helps to build your own,
Expand Down Expand Up @@ -693,56 +693,6 @@ Same as above but uses python 3.7.
</a>
</div>

Building Production images for 1.10 Airflow versions
----------------------------------------------------

With Breeze you can also use the master Dockerfile to build custom images for released Airflow versions.
This works in the same way as building production image from master, but you need to add additional switch
``--install-airflow-version``. You should pass version of airflow (as released in PyPI). It can be used
to install both released versions and release candidates. Similarly as in case of master images,
we can pass additional extras/dependencies to install via the additional flags.

.. code-block:: bash
./breeze build-image --production-image --additional-extras "jira" --install-airflow-version="1.10.11"
Builds airflow image with released Airflow version 1.10.11 and additional extra "jira" added.

.. code-block:: bash
./breeze build-image --production-image --install-airflow-version="1.10.11rc2"
Builds airflow image with released Airflow version 1.10.11rc2.


You can also build airflow directly from GitHub source code - by providing Git Reference via
``--install-airflow-reference``. The reference can be a branch name, tag name, or commit hash. This
is useful mostly for testing.

.. code-block:: bash
./breeze build-image --production-image --install-airflow-reference="v1-10-test"
This Builds airflow image from the current ``v1-10-test`` branch of Airflow.

.. code-block:: bash
./breeze build-image --production-image \
--install-airflow-reference="0d91fcf725f69e10f0969ca36f9e38e1d74110d0"
This Builds airflow image from the ``0d91fcf725f69e10f0969ca36f9e38e1d74110d0`` commit hash on
GitHub.

.. raw:: html

<div align="center">
<a href="https://youtu.be/4MCTXq-oF68?t=1586">
<img src="images/breeze/overlayed_breeze_build_images_released_versions.png" width="640"
alt="Airflow Breeze - Building Production images for 1.10 Airflow versions">
</a>
</div>


Running static checks
---------------------

Expand Down Expand Up @@ -823,8 +773,7 @@ Generating constraints
----------------------

Whenever setup.py gets modified, the CI master job will re-generate constraint files. Those constraint
files are stored in separated orphan branches: ``constraints-master``, ``constraints-2-0``
and ``constraints-1-10``.
files are stored in separated orphan branches: ``constraints-master``, ``constraints-2-0``.

Those are constraint files as described in detail in the
`<CONTRIBUTING.rst#pinned-constraint-files>`_ contributing documentation.
Expand Down Expand Up @@ -1319,21 +1268,18 @@ This is the current syntax for `./breeze <./breeze>`_:
-p, --python PYTHON_MAJOR_MINOR_VERSION
Python version used for the image. This is always major/minor version.
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
--install-airflow-version or --install-airflow-reference flags.
One of:
2.7 3.5 3.6 3.7 3.8
3.6 3.7 3.8
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
Uses differen version of Airflow when building PROD image.
2.0.3 2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
2.0.3 2.0.2 2.0.1 2.0.0 wheel sdist
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
Installs Airflow directly from reference in GitHub when building PROD image.
This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
This can be a GitHub branch like master or v2-0-test, or a tag like 2.0.0a1.
--installation-method INSTALLATION_METHOD
Method of installing Airflow in PROD image - either from the sources ('.')
Expand Down Expand Up @@ -1564,12 +1510,9 @@ This is the current syntax for `./breeze <./breeze>`_:
-p, --python PYTHON_MAJOR_MINOR_VERSION
Python version used for the image. This is always major/minor version.
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
--install-airflow-version or --install-airflow-reference flags.
One of:
2.7 3.5 3.6 3.7 3.8
3.6 3.7 3.8
-I, --production-image
Use production image for entering the environment and builds (not for tests).
Expand Down Expand Up @@ -1611,7 +1554,7 @@ This is the current syntax for `./breeze <./breeze>`_:
Generates pinned constraint files with all extras from setup.py. Those files are generated in
files folder - separate files for different python version. Those constraint files when
pushed to orphan constraints-master, constraints-2-0 and constraints-1-10 branches are used
pushed to orphan constraints-master, constraints-2-0 branches are used
to generate repeatable CI builds as well as run repeatable production image builds and
upgrades when you want to include installing or updating some of the released providers
released at the time particular airflow version was released. You can use those
Expand All @@ -1634,12 +1577,9 @@ This is the current syntax for `./breeze <./breeze>`_:
-p, --python PYTHON_MAJOR_MINOR_VERSION
Python version used for the image. This is always major/minor version.
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
--install-airflow-version or --install-airflow-reference flags.
One of:
2.7 3.5 3.6 3.7 3.8
3.6 3.7 3.8
-v, --verbose
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
Expand Down Expand Up @@ -1760,12 +1700,9 @@ This is the current syntax for `./breeze <./breeze>`_:
-p, --python PYTHON_MAJOR_MINOR_VERSION
Python version used for the image. This is always major/minor version.
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
--install-airflow-version or --install-airflow-reference flags.
One of:
2.7 3.5 3.6 3.7 3.8
3.6 3.7 3.8
####################################################################################################
Expand Down Expand Up @@ -1848,16 +1785,13 @@ This is the current syntax for `./breeze <./breeze>`_:
airflow is just removed. In this case airflow package should be added to dist folder
and --use-packages-from-dist flag should be used.
2.0.3 2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
2.0.3 2.0.2 2.0.1 2.0.0 wheel sdist none
--use-packages-from-dist
In CI image, if specified it will look for packages placed in dist folder and
it will install the packages after entering the image.
This is useful for testing provider packages.
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
--load-example-dags
Include Airflow example dags.
Expand Down Expand Up @@ -1950,12 +1884,9 @@ This is the current syntax for `./breeze <./breeze>`_:
-p, --python PYTHON_MAJOR_MINOR_VERSION
Python version used for the image. This is always major/minor version.
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
--install-airflow-version or --install-airflow-reference flags.
One of:
2.7 3.5 3.6 3.7 3.8
3.6 3.7 3.8
-b, --backend BACKEND
Backend to use for tests - it determines which database is used.
Expand Down Expand Up @@ -2017,12 +1948,9 @@ This is the current syntax for `./breeze <./breeze>`_:
-p, --python PYTHON_MAJOR_MINOR_VERSION
Python version used for the image. This is always major/minor version.
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
--install-airflow-version or --install-airflow-reference flags.
One of:
2.7 3.5 3.6 3.7 3.8
3.6 3.7 3.8
-F, --force-build-images
Forces building of the local docker images. The images are rebuilt
Expand Down Expand Up @@ -2419,12 +2347,9 @@ This is the current syntax for `./breeze <./breeze>`_:
-p, --python PYTHON_MAJOR_MINOR_VERSION
Python version used for the image. This is always major/minor version.
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
--install-airflow-version or --install-airflow-reference flags.
One of:
2.7 3.5 3.6 3.7 3.8
3.6 3.7 3.8
****************************************************************************************************
Choose backend to run for Airflow
Expand Down Expand Up @@ -2556,11 +2481,11 @@ This is the current syntax for `./breeze <./breeze>`_:
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
Uses differen version of Airflow when building PROD image.
2.0.3 2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
2.0.3 2.0.2 2.0.1 2.0.0 wheel sdist
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
Installs Airflow directly from reference in GitHub when building PROD image.
This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
This can be a GitHub branch like master or v2-0-test, or a tag like 2.0.0a1.
--installation-method INSTALLATION_METHOD
Method of installing Airflow in PROD image - either from the sources ('.')
Expand All @@ -2584,16 +2509,13 @@ This is the current syntax for `./breeze <./breeze>`_:
airflow is just removed. In this case airflow package should be added to dist folder
and --use-packages-from-dist flag should be used.
2.0.3 2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
2.0.3 2.0.2 2.0.1 2.0.0 wheel sdist none
--use-packages-from-dist
In CI image, if specified it will look for packages placed in dist folder and
it will install the packages after entering the image.
This is useful for testing provider packages.
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
****************************************************************************************************
Credentials
Expand Down
4 changes: 2 additions & 2 deletions CI.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ CI Environment
==============

Continuous Integration is important component of making Apache Airflow robust and stable. We are running
a lot of tests for every pull request, for master and v1-10-test branches and regularly as CRON jobs.
a lot of tests for every pull request, for master and v2-0-test branches and regularly as CRON jobs.

Our execution environment for CI is `GitHub Actions <https://github.com/features/actions>`_. GitHub Actions
(GA) are very well integrated with GitHub code and Workflow and it has evolved fast in 2019/202 to become
Expand Down Expand Up @@ -307,7 +307,7 @@ You can use those variables when you try to reproduce the build locally.
| | | | | tested set of dependency constraints |
| | | | | stored in separated "orphan" branches |
| | | | | of the airflow repository |
| | | | | ("constraints-master, "constraints-1-10") |
| | | | | ("constraints-master, "constraints-2-0") |
| | | | | but when this flag is set to anything but false |
| | | | | (for example commit SHA), they are not used |
| | | | | used and "eager" upgrade strategy is used |
Expand Down
5 changes: 2 additions & 3 deletions IMAGES.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,8 @@ The images are named as follows:
where:

* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``,
``v2-0-test``, ``v1-10-test``, ``2.0.0``. The ``master``, ``v1-10-test`` ``v2-0-test`` labels are
built from branches so they change over time. The ``1.10.*`` and ``2.*`` labels are built from git tags
``v2-0-test``, ``2.0.0``. The ``master``, ``v2-0-test`` labels are
built from branches so they change over time. The ````2.*`` labels are built from git tags
and they are "fixed" once built.
* ``PYTHON_MAJOR_MINOR_VERSION`` - version of python used to build the image. Examples: ``3.6``, ``3.7``,
``3.8``
Expand Down Expand Up @@ -562,7 +562,6 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | used. By default it is set to |
| | | ``constraints-master`` but can be |
| | | ``constraints-2-0`` for 2.0.* versions |
| | | ``constraints-1-10`` for 1.10.* versions |
| | | or it could point to specific version |
| | | for example ``constraints-2.0.0`` |
| | | is empty, it is auto-detected |
Expand Down
5 changes: 2 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,9 +137,8 @@ our dependencies as open as possible (in `setup.py`) so users can install differ
if needed. This means that from time to time plain `pip install apache-airflow` will not work or will
produce unusable Airflow installation.

In order to have repeatable installation, however, introduced in **Airflow 1.10.10** and updated in
**Airflow 1.10.12** we also keep a set of "known-to-be-working" constraint files in the
orphan `constraints-master`, `constraints-2-0` and `constraints-1-10` branches. We keep those "known-to-be-working"
In order to have repeatable installation, however, we also keep a set of "known-to-be-working" constraint
files in the orphan `constraints-master`, `constraints-2-0` branches. We keep those "known-to-be-working"
constraints files separately per major/minor Python version.
You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify
correct Airflow tag/version/branch and Python versions in the URL.
Expand Down
2 changes: 1 addition & 1 deletion airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ spec:
- name: GIT_SYNC_REV
value: "HEAD"
- name: GIT_SYNC_BRANCH
value: "v1-10-stable"
value: "v2-0-stable"
- name: GIT_SYNC_REPO
value: "https://github.com/apache/airflow.git"
- name: GIT_SYNC_DEPTH
Expand Down
Loading

0 comments on commit c4476d2

Please sign in to comment.