Skip to content

Commit

Permalink
Bump Airflow Version to 1.10.14
Browse files Browse the repository at this point in the history
  • Loading branch information
kaxil committed Dec 3, 2020
1 parent f46ed7c commit fc6d0a8
Show file tree
Hide file tree
Showing 5 changed files with 45 additions and 45 deletions.
18 changes: 9 additions & 9 deletions IMAGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ The images are named as follows:

where:

* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``, ``v1-10-test``, ``1.10.13``
* ``BRANCH_OR_TAG`` - branch or tag used when creating the image. Examples: ``master``, ``v1-10-test``, ``1.10.14``
The ``master`` and ``v1-10-test`` labels are built from branches so they change over time. The ``1.10.*`` and in
the future ``2.*`` labels are build from git tags and they are "fixed" once built.
* ``PYTHON_MAJOR_MINOR_VERSION`` - version of python used to build the image. Examples: ``3.5``, ``3.7``
Expand Down Expand Up @@ -115,15 +115,15 @@ parameter to Breeze:
.. code-block:: bash
./breeze build-image --python 3.7 --additional-extras=presto \
--production-image --install-airflow-version=1.10.13
--production-image --install-airflow-version=1.10.14
This will build the image using command similar to:

.. code-block:: bash
pip install \
apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.13 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.6.txt"
apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.14 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
.. note::
On 30th of November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver.
Expand Down Expand Up @@ -218,8 +218,8 @@ For example:
apache/airflow:master-python3.6 - production "latest" image from current master
apache/airflow:master-python3.6-ci - CI "latest" image from current master
apache/airflow:v1-10-test-python2.7-ci - CI "latest" image from current v1-10-test branch
apache/airflow:1.10.13-python3.6 - production image for 1.10.13 release
apache/airflow:1.10.13-1-python3.6 - production image for 1.10.13 with some patches applied
apache/airflow:1.10.14-python3.6 - production image for 1.10.14 release
apache/airflow:1.10.14-1-python3.6 - production image for 1.10.14 with some patches applied
You can see DockerHub images at `<https://hub.docker.com/repository/docker/apache/airflow>`_
Expand Down Expand Up @@ -300,7 +300,7 @@ additional apt dev and runtime dependencies.
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
Expand All @@ -316,7 +316,7 @@ the same image can be built using ``breeze`` (it supports auto-completion of the
.. code-block:: bash
./breeze build-image -f Dockerfile.ci \
--production-image --python 3.7 --install-airflow-version=1.10.13 \
--production-image --python 3.7 --install-airflow-version=1.10.14 \
--additional-extras=jdbc --additional-python-deps="pandas" \
--additional-dev-apt-deps="gcc g++" --additional-runtime-apt-deps="default-jre-headless"
You can build the default production image with standard ``docker build`` command but they will only build
Expand All @@ -334,7 +334,7 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Airflow is not a streaming solution, but it is often used to process real-time d

Apache Airflow is tested with:

| | Master version (2.0.0dev) | Stable version (1.10.13) |
| | Master version (2.0.0dev) | Stable version (1.10.14) |
| ------------ | ------------------------- | ------------------------ |
| Python | 3.6, 3.7, 3.8 | 2.7, 3.5, 3.6, 3.7, 3.8 |
| PostgreSQL | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 |
Expand Down Expand Up @@ -109,7 +109,7 @@ if needed. This means that from time to time plain `pip install apache-airflow`
produce unusable Airflow installation.

In order to have repeatable installation, however, introduced in **Airflow 1.10.10** and updated in
**Airflow 1.10.13** we also keep a set of "known-to-be-working" constraint files in the
**Airflow 1.10.12** we also keep a set of "known-to-be-working" constraint files in the
orphan `constraints-master` and `constraints-1-10` branches. We keep those "known-to-be-working"
constraints files separately per major/minor python version.
You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify
Expand All @@ -118,8 +118,8 @@ correct Airflow tag/version/branch and python versions in the URL.
1. Installing just Airflow:

```bash
pip install apache-airflow==1.10.13 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.7.txt"
pip install apache-airflow==1.10.14 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt"
```

**NOTE!!!**
Expand All @@ -133,8 +133,8 @@ pip to version 20.2.4 `pip upgrade --pip==20.2.4` or, in case you use Pip 20.3,

2. Installing with extras (for example postgres,gcp)
```bash
pip install apache-airflow[postgres,gcp]==1.10.13 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.7.txt"
pip install apache-airflow[postgres,gcp]==1.10.14 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt"
```

For information on installing backport providers check https://airflow.readthedocs.io/en/latest/backport-providers.html.
Expand Down
2 changes: 1 addition & 1 deletion airflow/version.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@
# under the License.
#

version = '1.10.13'
version = '1.10.14'
8 changes: 4 additions & 4 deletions docs/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ if needed. This means that from time to time plain ``pip install apache-airflow`
produce unusable Airflow installation.

In order to have repeatable installation, however, starting from **Airflow 1.10.10** and updated in
**Airflow 1.10.13** we also keep a set of "known-to-be-working" constraint files in the
**Airflow 1.10.12** we also keep a set of "known-to-be-working" constraint files in the
``constraints-master`` and ``constraints-1-10`` orphan branches.
Those "known-to-be-working" constraints are per major/minor python version. You can use them as constraint
files when installing Airflow from PyPI. Note that you have to specify correct Airflow version
Expand All @@ -51,11 +51,11 @@ and python versions in the URL.

.. code-block:: bash
AIRFLOW_VERSION=1.10.13
AIRFLOW_VERSION=1.10.14
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-1.10.13/constraints-3.6.txt
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
Expand All @@ -70,7 +70,7 @@ and python versions in the URL.

.. code-block:: bash
AIRFLOW_VERSION=1.10.13
AIRFLOW_VERSION=1.10.14
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
Expand Down
50 changes: 25 additions & 25 deletions docs/production-deployment.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ You should be aware, about a few things:

.. code-block:: dockerfile
FROM: apache/airflow:1.10.13
FROM: apache/airflow:1.10.14
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
Expand All @@ -81,7 +81,7 @@ You should be aware, about a few things:

.. code-block:: dockerfile
FROM: apache/airflow:1.10.13
FROM: apache/airflow:1.10.14
RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add
Expand All @@ -92,7 +92,7 @@ You should be aware, about a few things:

.. code-block:: dockerfile
FROM: apache/airflow:1.10.13
FROM: apache/airflow:1.10.14
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
Expand Down Expand Up @@ -125,7 +125,7 @@ in the `<#production-image-build-arguments>`_ chapter below.

Here just a few examples are presented which should give you general understanding of what you can customize.

This builds the production image in version 3.7 with additional airflow extras from 1.10.13 PyPI package and
This builds the production image in version 3.7 with additional airflow extras from 1.10.14 PyPI package and
additional apt dev and runtime dependencies.

.. code-block:: bash
Expand All @@ -134,7 +134,7 @@ additional apt dev and runtime dependencies.
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
Expand All @@ -150,7 +150,7 @@ the same image can be built using ``breeze`` (it supports auto-completion of the
.. code-block:: bash
./breeze build-image \
--production-image --python 3.7 --install-airflow-version=1.10.13 \
--production-image --python 3.7 --install-airflow-version=1.10.14 \
--additional-extras=jdbc --additional-python-deps="pandas" \
--additional-dev-apt-deps="gcc g++" --additional-runtime-apt-deps="default-jre-headless"
Expand All @@ -166,7 +166,7 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
Expand Down Expand Up @@ -225,15 +225,15 @@ Preparing the constraint files and wheel files:
pip download --dest docker-context-files \
--constraint docker-context-files/constraints-1-10.txt \
apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv]==1.10.13
apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv]==1.10.14
Building the image (after copying the files downloaded to the "docker-context-files" directory:

.. code-block:: bash
./breeze build-image \
--production-image --python 3.7 --install-airflow-version=1.10.13 \
--production-image --python 3.7 --install-airflow-version=1.10.14 \
--disable-mysql-client-installation --disable-pip-cache --add-local-pip-wheels \
--constraints-location="/docker-context-files/constraints-1-10.txt"
Expand All @@ -245,7 +245,7 @@ or
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
Expand Down Expand Up @@ -392,7 +392,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | ``constraints-master`` but can be |
| | | ``constraints-1-10`` for 1.10.* versions |
| | | or it could point to specific version |
| | | for example ``constraints-1.10.13`` |
| | | for example ``constraints-1.10.14`` |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_EXTRAS`` | (see Dockerfile) | Default extras with which airflow is |
| | | installed |
Expand Down Expand Up @@ -503,7 +503,7 @@ production image. There are three types of build:
| ``AIRFLOW_INSTALL_VERSION`` | Optional - might be used for |
| | package installation case to |
| | set Airflow version for example |
| | "==1.10.13" |
| | "==1.10.14" |
+-----------------------------------+-----------------------------------+
| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | reference (branch or tag) from |
| | GitHub where constraints file |
Expand All @@ -512,7 +512,7 @@ production image. There are three types of build:
| | ``constraints-1-10`` for 1.10.* |
| | constraint or if you want to |
| | point to specific version |
| | might be ``constraints-1.10.13`` |
| | might be ``constraints-1.10.14`` |
+-----------------------------------+-----------------------------------+
| ``SLUGIFY_USES_TEXT_UNIDECODE`` | In case of of installing airflow |
| | 1.10.2 or 1.10.1 you need to |
Expand Down Expand Up @@ -546,22 +546,22 @@ of 2.0 currently):
docker build .
This builds the production image in version 3.7 with default extras from 1.10.13 tag and
This builds the production image in version 3.7 with default extras from 1.10.14 tag and
constraints taken from constraints-1-10-12 branch in GitHub.

.. code-block:: bash
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="https://github.com/apache/airflow/archive/1.10.13.tar.gz#egg=apache-airflow" \
--build-arg AIRFLOW_INSTALL_SOURCES="https://github.com/apache/airflow/archive/1.10.14.tar.gz#egg=apache-airflow" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_BRANCH="v1-10-test" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty"
This builds the production image in version 3.7 with default extras from 1.10.13 PyPI package and
constraints taken from 1.10.13 tag in GitHub and pre-installed pip dependencies from the top
This builds the production image in version 3.7 with default extras from 1.10.14 PyPI package and
constraints taken from 1.10.14 tag in GitHub and pre-installed pip dependencies from the top
of v1-10-test branch.

.. code-block:: bash
Expand All @@ -570,30 +570,30 @@ of v1-10-test branch.
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_BRANCH="v1-10-test" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.13" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.14" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty"
This builds the production image in version 3.7 with additional airflow extras from 1.10.13 PyPI package and
additional python dependencies and pre-installed pip dependencies from 1.10.13 tagged constraints.
This builds the production image in version 3.7 with additional airflow extras from 1.10.14 PyPI package and
additional python dependencies and pre-installed pip dependencies from 1.10.14 tagged constraints.

.. code-block:: bash
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_BRANCH="v1-10-test" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.13" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1.10.14" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs"
--build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client"
This builds the production image in version 3.7 with additional airflow extras from 1.10.13 PyPI package and
This builds the production image in version 3.7 with additional airflow extras from 1.10.14 PyPI package and
additional apt dev and runtime dependencies.

.. code-block:: bash
Expand All @@ -602,7 +602,7 @@ additional apt dev and runtime dependencies.
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
--build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.13" \
--build-arg AIRFLOW_INSTALL_VERSION="==1.10.14" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-1-10" \
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
Expand Down

0 comments on commit fc6d0a8

Please sign in to comment.