Attempt to add Python 3.11 support

Python 3.11 has been released as scheduled on October 25, 2022 and this is the first attempt to see how far Airflow (mostly dependencies) are from being ready to officially support 3.11.
apache · Oct 25, 2022 · c2dd81c · c2dd81c
1 parent 7fcdb49
commit c2dd81c
Show file tree

Hide file tree

Showing 50 changed files with 2,414 additions and 2,361 deletions.
diff --git a/CI.rst b/CI.rst
@@ -59,7 +59,7 @@ Container Registry used as cache
 We are using GitHub Container Registry to store the results of the ``Build Images``
 workflow which is used in the ``Tests`` workflow.
 
-Currently in main version of Airflow we run tests in 4 different versions of Python (3.7, 3.8, 3.9, 3.10)
+Currently in main version of Airflow we run tests in 4 different versions of Python (3.7, 3.8, 3.9, 3.10, 3.11)
 which means that we have to build 8 images (4 CI ones and 4 PROD ones). Yet we run around 12 jobs
 with each of the CI images. That is a lot of time to just build the environment to run. Therefore
 we are utilising ``pull_request_target`` feature of GitHub Actions.

diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst
@@ -51,7 +51,7 @@ Required Software Packages
 Use system-level package managers like yum, apt-get for Linux, or
 Homebrew for macOS to install required software packages:
 
-* Python (One of: 3.7, 3.8, 3.9, 3.10)
+* Python (One of: 3.7, 3.8, 3.9, 3.10, 3.11)
 * MySQL 5.7+
 * libxml
 
@@ -102,7 +102,7 @@ Creating a Local virtualenv
 
 To use your IDE for Airflow development and testing, you need to configure a virtual
 environment. Ideally you should set up virtualenv for all Python versions that Airflow
-supports (3.7, 3.8, 3.9, 3.10).
+supports (3.7, 3.8, 3.9, 3.10, 3.11).
 
 To create and initialize the local virtualenv:
 
@@ -122,7 +122,7 @@ To create and initialize the local virtualenv:
 
     .. code-block:: bash
 
-      conda create -n airflow python=3.7  # or 3.8, 3.9, 3.10
+      conda create -n airflow python=3.7  # or 3.8, 3.9, 3.10, 3.11
       conda activate airflow
 
 2. Install Python PIP requirements:
@@ -150,7 +150,7 @@ for different python versions). For development on current main source:
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     pip install -e ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 
@@ -163,7 +163,7 @@ You can also install Airflow in non-editable mode:
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     pip install ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 
@@ -173,7 +173,7 @@ sources, unless you set ``INSTALL_PROVIDERS_FROM_SOURCES`` environment variable
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     INSTALL_PROVIDERS_FROM_SOURCES="true" pip install ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 

diff --git a/README.md b/README.md
@@ -87,7 +87,7 @@ Apache Airflow is tested with:
 
 |                     | Main version (dev)           | Stable version (2.4.2)       |
 |---------------------|------------------------------|------------------------------|
-| Python              | 3.7, 3.8, 3.9, 3.10          | 3.7, 3.8, 3.9, 3.10          |
+| Python              | 3.7, 3.8, 3.9, 3.10, 3.11    | 3.7, 3.8, 3.9, 3.10, 3.11    |
 | Platform            | AMD64/ARM64(\*)              | AMD64/ARM64(\*)              |
 | Kubernetes          | 1.21, 1.22, 1.23, 1.24, 1.25 | 1.21, 1.22, 1.23, 1.24, 1.25 |
 | PostgreSQL          | 10, 11, 12, 13, 14           | 10, 11, 12, 13, 14           |

diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
@@ -985,7 +985,7 @@ the older branches, you should set the "skip" field to true.
 ## Verify production images
 
 ```shell script
-for PYTHON in 3.7 3.8 3.9 3.10
+for PYTHON in 3.7 3.8 3.9 3.10 3.11
 do
     docker pull apache/airflow:${VERSION}-python${PYTHON}
     breeze prod-image verify --image-name apache/airflow:${VERSION}-python${PYTHON}

diff --git a/dev/breeze/SELECTIVE_CHECKS.md b/dev/breeze/SELECTIVE_CHECKS.md
@@ -109,8 +109,8 @@ The selective check outputs available are described below:
 
 | Output                             | Meaning of the output                                                                                  | Example value                                                 |
 |------------------------------------|--------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|
-| all-python-versions                | List of all python versions there are available in the form of JSON array                              | ['3.7', '3.8', '3.9', '3.10']                                 |
-| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string                  | 3.7 3.8 3.9 3.10                                              |
+| all-python-versions                | List of all python versions there are available in the form of JSON array                              | ['3.7', '3.8', '3.9', '3.10', '3.11']                         |
+| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string                  | 3.7 3.8 3.9 3.10 3.11                                         |
 | basic-checks-only                  | Whether to run all static checks ("false") or only basic set of static checks ("true")                 | false                                                         |
 | cache-directive                    | Which cache should be be used for images ("registry", "local" , "disabled")                            | registry                                                      |
 | debug-resources                    | Whether resources usage should be printed during parallel job execution ("true"/ "false")              | false                                                         |

diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -37,7 +37,7 @@
 APACHE_AIRFLOW_GITHUB_REPOSITORY = "apache/airflow"
 
 # Checked before putting in build cache
-ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 DEFAULT_PYTHON_MAJOR_MINOR_VERSION = ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS[0]
 ALLOWED_ARCHITECTURES = [Architecture.X86_64, Architecture.ARM]
 ALLOWED_BACKENDS = ["sqlite", "mysql", "postgres", "mssql"]
@@ -153,7 +153,7 @@ def get_default_platform_machine() -> str:
 PYTHONDONTWRITEBYTECODE = True
 
 PRODUCTION_IMAGE = False
-ALL_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALL_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 CURRENT_PYTHON_MAJOR_MINOR_VERSIONS = ALL_PYTHON_MAJOR_MINOR_VERSIONS
 CURRENT_POSTGRES_VERSIONS = ["10", "11", "12", "13", "14"]
 DEFAULT_POSTGRES_VERSION = CURRENT_POSTGRES_VERSIONS[0]

diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -366,7 +366,7 @@ def mysql_exclude(self) -> list[dict[str, str]]:
 
     @cached_property
     def sqlite_exclude(self) -> list[dict[str, str]]:
-        return [{"python-version": "3.9"}] if self._full_tests_needed else []
+        return [{"python-version": "3.11"}] if self._full_tests_needed else []
 
     @cached_property
     def kubernetes_versions(self) -> list[str]:

diff --git a/dev/breeze/tests/test_cache.py b/dev/breeze/tests/test_cache.py
@@ -36,8 +36,8 @@
     [
         ("backend", "mysql", (True, ["sqlite", "mysql", "postgres", "mssql"]), None),
         ("backend", "xxx", (False, ["sqlite", "mysql", "postgres", "mssql"]), None),
-        ("python_major_minor_version", "3.8", (True, ["3.7", "3.8", "3.9", "3.10"]), None),
-        ("python_major_minor_version", "3.5", (False, ["3.7", "3.8", "3.9", "3.10"]), None),
+        ("python_major_minor_version", "3.8", (True, ["3.7", "3.8", "3.9", "3.10", "3.11"]), None),
+        ("python_major_minor_version", "3.5", (False, ["3.7", "3.8", "3.9", "3.10", "3.11"]), None),
         ("missing", "value", None, AttributeError),
     ],
 )

diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
@@ -206,8 +206,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
             pytest.param(
                 ("setup.py",),
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "needs-helm-tests": "true",
                     "run-tests": "true",
@@ -223,8 +223,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
             pytest.param(
                 ("generated/provider_dependencies.json",),
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "needs-helm-tests": "true",
                     "run-tests": "true",
@@ -260,8 +260,8 @@ def test_expected_output_pull_request_main(
                 ("full tests needed",),
                 "main",
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -280,8 +280,8 @@ def test_expected_output_pull_request_main(
                 ),
                 "main",
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -298,8 +298,8 @@ def test_expected_output_pull_request_main(
                 ("full tests needed",),
                 "main",
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -316,8 +316,8 @@ def test_expected_output_pull_request_main(
                 ("full tests needed",),
                 "v2-3-stable",
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -513,8 +513,8 @@ def test_expected_output_pull_request_target(
             (),
             "main",
             {
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "true",
                 "run-tests": "true",
@@ -529,8 +529,8 @@ def test_expected_output_pull_request_target(
             (),
             "v2-3-stable",
             {
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "false",
                 "run-tests": "true",
@@ -546,8 +546,8 @@ def test_expected_output_pull_request_target(
             (),
             "main",
             {
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "true",
                 "run-tests": "true",
@@ -595,8 +595,8 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(github_event):
     )
     assert_outputs_are_printed(
         {
-            "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-            "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+            "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+            "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
             "image-build": "true",
             "needs-helm-tests": "true",
             "run-tests": "true",

diff --git a/dev/provider_packages/prepare_provider_packages.py b/dev/provider_packages/prepare_provider_packages.py
@@ -53,7 +53,7 @@
 from rich.syntax import Syntax
 from yaml import safe_load
 
-ALL_PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALL_PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 
 INITIAL_CHANGELOG_CONTENT = """
 

diff --git a/dev/retag_docker_images.py b/dev/retag_docker_images.py
@@ -31,7 +31,7 @@
 
 import rich_click as click
 
-PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 
 GHCR_IO_PREFIX = "ghcr.io"
 

diff --git a/docs/apache-airflow/installation/prerequisites.rst b/docs/apache-airflow/installation/prerequisites.rst
@@ -20,7 +20,7 @@ Prerequisites
 
 Starting with Airflow 2.3.0, Airflow is tested with:.
 
-* Python: 3.7, 3.8, 3.9, 3.10
+* Python: 3.7, 3.8, 3.9, 3.10, 3.11
 
 * Databases:
 

diff --git a/docs/apache-airflow/upgrading-from-1-10/index.rst b/docs/apache-airflow/upgrading-from-1-10/index.rst
@@ -34,7 +34,7 @@ Airflow 1.10 was the last release series to support Python 2. Airflow 2.0.0
 requires Python 3.6+ and has been tested with Python versions 3.6, 3.7 and 3.8.
 Python 3.9 support was added from Airflow 2.1.2.
 
-Airflow 2.3.0 dropped support for Python 3.6. It's tested with Python 3.7, 3.8, 3.9 and 3.10.
+Airflow 2.3.0 dropped support for Python 3.6. It's tested with Python 3.7, 3.8, 3.9, 3.10, 3.11.
 
 If you have a specific task that still requires Python 2 then you can use the ``@task.virtualenv``, ``@task.docker`` or ``@task.kubernetes`` decorators for this.
 

diff --git a/docs/docker-stack/build.rst b/docs/docker-stack/build.rst
@@ -262,14 +262,14 @@ There are two types of images you can extend your image from:
 
 Naming conventions for the images:
 
-+----------------+------------------+---------------------------------+--------------------------------------+
-| Image          | Python           | Standard image                  | Slim image                           |
-+================+==================+=================================+======================================+
-| Latest default | 3.7              | apache/airflow:latest           | apache/airflow:slim-latest           |
-| Default        | 3.7              | apache/airflow:X.Y.Z            | apache/airflow:slim-X.Y.Z            |
-| Latest         | 3.7,3.8,3.9,3.10 | apache/airflow:latest-pythonN.M | apache/airflow:slim-latest-pythonN.M |
-| Specific       | 3.7,3.8,3.9,3.10 | apache/airflow:X.Y.Z-pythonN.M  | apache/airflow:slim-X.Y.Z-pythonN.M  |
-+----------------+------------------+---------------------------------+--------------------------------------+
++----------------+-----------------------+---------------------------------+--------------------------------------+
+| Image          | Python                | Standard image                  | Slim image                           |
++================+=======================+=================================+======================================+
+| Latest default | 3.7                   | apache/airflow:latest           | apache/airflow:slim-latest           |
+| Default        | 3.7                   | apache/airflow:X.Y.Z            | apache/airflow:slim-X.Y.Z            |
+| Latest         | 3.7,3.8,3.9,3.10,3.11 | apache/airflow:latest-pythonN.M | apache/airflow:slim-latest-pythonN.M |
+| Specific       | 3.7,3.8,3.9,3.10,3.11 | apache/airflow:X.Y.Z-pythonN.M  | apache/airflow:slim-X.Y.Z-pythonN.M  |
++----------------+-----------------------+---------------------------------+--------------------------------------+
 
 * The "latest" image is always the latest released stable version available.