From b97422459c643ad187bc4d886ed96450b4685bf1 Mon Sep 17 00:00:00 2001 From: Xiaodong DENG Date: Fri, 13 Jun 2025 16:55:06 -0700 Subject: [PATCH] Update the executor and provider doc to highlight the two statically coded hybrid executors are no longer supported in Airflow 3.0.0+ --- airflow-core/docs/core-concepts/executor/index.rst | 8 ++++++-- providers/celery/docs/celery_kubernetes_executor.rst | 5 +++++ .../cncf/kubernetes/docs/local_kubernetes_executor.rst | 6 ++++++ 3 files changed, 17 insertions(+), 2 deletions(-) diff --git a/airflow-core/docs/core-concepts/executor/index.rst b/airflow-core/docs/core-concepts/executor/index.rst index 33427ff2bcf23..ef488f4d9f8e2 100644 --- a/airflow-core/docs/core-concepts/executor/index.rst +++ b/airflow-core/docs/core-concepts/executor/index.rst @@ -112,6 +112,8 @@ Airflow tasks are executed ad hoc inside containers/pods. Each task is isolated New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending on the executor selected. +.. _using-multiple-executors-concurrently: + Using Multiple Executors Concurrently ------------------------------------- @@ -208,11 +210,13 @@ Logging works the same as the single executor use case. Statically-coded Hybrid Executors ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -There are currently two "statically coded" executors, these executors are hybrids of two different executors: the :doc:`LocalKubernetesExecutor ` and the :doc:`CeleryKubernetesExecutor `. Their implementation is not native or intrinsic to core Airflow. These hybrid executors instead make use of the ``queue`` field on Task Instances to indicate and persist which sub-executor to run on. This is a misuse of the ``queue`` field and makes it impossible to use it for its intended purpose when using these hybrid executors. +There were two "statically coded" executors, but they are no longer supported starting from Airflow 3.0.0. + +These executors are hybrids of two different executors: the :doc:`LocalKubernetesExecutor ` and the :doc:`CeleryKubernetesExecutor `. Their implementation is not native or intrinsic to core Airflow. These hybrid executors instead make use of the ``queue`` field on Task Instances to indicate and persist which sub-executor to run on. This is a misuse of the ``queue`` field and makes it impossible to use it for its intended purpose when using these hybrid executors. Executors such as these also require hand crafting new "concrete" classes to create each permutation of possible combinations of executors. This is untenable as more executors are created and leads to more maintenance overhead. Bespoke coding effort should not be required to use any combination of executors. -Therefore using these types of executors is no longer recommended. +Therefore using these types of executors is no longer supported starting from Airflow 3.0.0. It's recommended to use the :ref:`Using Multiple Executors Concurrently ` feature instead. Writing Your Own Executor diff --git a/providers/celery/docs/celery_kubernetes_executor.rst b/providers/celery/docs/celery_kubernetes_executor.rst index 77a6a81681266..2ad5dbca8c5e0 100644 --- a/providers/celery/docs/celery_kubernetes_executor.rst +++ b/providers/celery/docs/celery_kubernetes_executor.rst @@ -25,6 +25,11 @@ CeleryKubernetes Executor ``apache-airflow-providers-cncf-kubernetes>=7.4.0`` or by installing Airflow with the ``celery`` and ``cncf.kubernetes`` extras: ``pip install 'apache-airflow[celery,cncf.kubernetes]'``. +.. note:: + + ``CeleryKubernetesExecutor`` is no longer supported starting from Airflow 3.0.0. You can use the + :ref:`Using Multiple Executors Concurrently ` feature instead, + which provides equivalent functionality in a more flexible manner. The :class:`~airflow.providers.celery.executors.celery_kubernetes_executor.CeleryKubernetesExecutor` allows users to run simultaneously a ``CeleryExecutor`` and a ``KubernetesExecutor``. diff --git a/providers/cncf/kubernetes/docs/local_kubernetes_executor.rst b/providers/cncf/kubernetes/docs/local_kubernetes_executor.rst index daed3d4d1f270..8f600418a9fb0 100644 --- a/providers/cncf/kubernetes/docs/local_kubernetes_executor.rst +++ b/providers/cncf/kubernetes/docs/local_kubernetes_executor.rst @@ -28,6 +28,12 @@ LocalKubernetes Executor or by installing Airflow with the ``cncf.kubernetes`` extras: ``pip install 'apache-airflow[cncf.kubernetes]'``. +.. note:: + + ``LocalKubernetesExecutor`` is no longer supported starting from Airflow 3.0.0. You can use the + :ref:`Using Multiple Executors Concurrently ` feature instead, + which provides equivalent functionality in a more flexible manner. + The :class:`~airflow.providers.cncf.kubernetes.executors.local_kubernetes_executor.LocalKubernetesExecutor` allows users to simultaneously run a ``LocalExecutor`` and a ``KubernetesExecutor``. An executor is chosen to run a task based on the task's queue.