From b99f64e389f125c33a3ca895f11cba30f6d93e29 Mon Sep 17 00:00:00 2001 From: N R Navaneet <156576749+nrnavaneet@users.noreply.github.com> Date: Tue, 22 Jul 2025 01:17:20 +0530 Subject: [PATCH] [v3-0-test] fix(systemd): Update systemd unit files and documentation for Airflow 3.0 (#52294) * docs: update public interface doc to reflect airflow.sdk and AIP-72 - Added a note under "Using Airflow Public Interfaces" to recommend using `airflow.sdk` as the official interface from Airflow 3.0. - Referenced AIP-72 and linked related documentation. - Encouraged users to prefer REST API and Python Client for integrations. * Update airflow-core/docs/public-airflow-interface.rst Great Co-authored-by: Jarek Potiuk * fix(systemd): Update systemd service files and docs for Airflow 3.0 - Updated systemd unit files to support Airflow 3.0 architecture. - Replaced deprecated `webserver` with `api-server` in ExecStart. - Added service units for new components: `triggerer`, `dag-processor`, and `api-server`. - Adjusted ExecStart commands to support virtualenv-based deployments. - Improved documentation (`run-with-systemd.rst`) to clarify service changes, virtualenv usage, and installation paths. - Linked to the updated GitHub `scripts/systemd` directory for latest service files. Tested Airflow 3.0 manually on macOS using virtualenv, confirming correct startup behavior for `scheduler`, `api-server`, and `triggerer`. * fix(systemd): Update systemd service files and docs for Airflow 3.0 * Minor test failure fixes --------- (cherry picked from commit 097ae4517d06d0fae1ae5890e206fffa7e64c148) Co-authored-by: N R Navaneet <156576749+nrnavaneet@users.noreply.github.com> Co-authored-by: Jarek Potiuk --- airflow-core/docs/howto/run-with-systemd.rst | 31 ++++++++++++++++++++ scripts/systemd/airflow-api.service | 31 ++++++++++++++++++++ scripts/systemd/airflow-flower.service | 1 + scripts/systemd/airflow-kerberos.service | 2 +- scripts/systemd/airflow-scheduler.service | 2 +- scripts/systemd/airflow-triggerer.service | 31 ++++++++++++++++++++ scripts/systemd/airflow-webserver.service | 2 +- scripts/systemd/airflow-worker.service | 2 +- 8 files changed, 98 insertions(+), 4 deletions(-) create mode 100644 scripts/systemd/airflow-api.service create mode 100644 scripts/systemd/airflow-triggerer.service diff --git a/airflow-core/docs/howto/run-with-systemd.rst b/airflow-core/docs/howto/run-with-systemd.rst index 92dafd5760cd4..1006ce4a06af6 100644 --- a/airflow-core/docs/howto/run-with-systemd.rst +++ b/airflow-core/docs/howto/run-with-systemd.rst @@ -27,6 +27,12 @@ In the ``scripts/systemd`` directory, you can find unit files that have been tested on Redhat based systems. These files can be used as-is by copying them over to ``/usr/lib/systemd/system``. +You can find the latest systemd unit files on GitHub: +https://github.com/apache/airflow/tree/main/scripts/systemd + +Assumptions +----------- + The following **assumptions** have been made while creating these unit files: #. Airflow runs as the following ``user:group`` ``airflow:airflow``. @@ -34,7 +40,32 @@ The following **assumptions** have been made while creating these unit files: If this is not the case, appropriate changes will need to be made. +Environment Configuration +------------------------- + Please **note** that environment configuration is picked up from ``/etc/sysconfig/airflow``. An example file is supplied within ``scripts/systemd``. You can also define configuration at :envvar:`AIRFLOW_HOME` or :envvar:`AIRFLOW_CONFIG`. + +Using Virtual Environments +-------------------------- + +.. note:: + If Airflow is installed inside a virtual environment (e.g. ``venv`` or ``conda``), you must update the ``ExecStart`` line in each systemd unit file to activate the virtualenv first. + + Example: + + .. code-block:: ini + + ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow scheduler' + + Replace ``/home/airflow/airflow_venv/`` with the path to your virtual environment. + +New Airflow 3.0 Services +------------------------ + +Since Apache Airflow 3.0, additional components have been split out into separate services. The following new unit files are available: + +- ``airflow-triggerer.service`` for deferrable task triggering +- ``airflow-api.service`` for the standalone REST API server diff --git a/scripts/systemd/airflow-api.service b/scripts/systemd/airflow-api.service new file mode 100644 index 0000000000000..6fc5ceb3ded11 --- /dev/null +++ b/scripts/systemd/airflow-api.service @@ -0,0 +1,31 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[Unit] +Description=Airflow API Server +After=network.target + +[Service] +Environment=AIRFLOW_HOME=/home/airflow/airflow_home +User=airflow +Group=airflow +Restart=always +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow api' + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/airflow-flower.service b/scripts/systemd/airflow-flower.service index 67946630e0fb3..4056dbf98a0e5 100644 --- a/scripts/systemd/airflow-flower.service +++ b/scripts/systemd/airflow-flower.service @@ -27,6 +27,7 @@ User=airflow Group=airflow Type=simple ExecStart=/bin/airflow celery flower +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow celery flower' Restart=on-failure RestartSec=5s diff --git a/scripts/systemd/airflow-kerberos.service b/scripts/systemd/airflow-kerberos.service index edb21dd04b938..549e3550536da 100644 --- a/scripts/systemd/airflow-kerberos.service +++ b/scripts/systemd/airflow-kerberos.service @@ -26,7 +26,7 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow kerberos +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && kerberos' Restart=on-failure RestartSec=5s diff --git a/scripts/systemd/airflow-scheduler.service b/scripts/systemd/airflow-scheduler.service index 371cdc6bf840d..ada93e10125c0 100644 --- a/scripts/systemd/airflow-scheduler.service +++ b/scripts/systemd/airflow-scheduler.service @@ -26,7 +26,7 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow scheduler +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow scheduler' Restart=always RestartSec=5s diff --git a/scripts/systemd/airflow-triggerer.service b/scripts/systemd/airflow-triggerer.service new file mode 100644 index 0000000000000..f7345aa2bea16 --- /dev/null +++ b/scripts/systemd/airflow-triggerer.service @@ -0,0 +1,31 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[Unit] +Description=Airflow Triggerer +After=network.target + +[Service] +Environment=AIRFLOW_HOME=/home/airflow/airflow_home +User=airflow +Group=airflow +Restart=always +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow triggerer' + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/airflow-webserver.service b/scripts/systemd/airflow-webserver.service index c27022a26895d..eddeeb43bc5f0 100644 --- a/scripts/systemd/airflow-webserver.service +++ b/scripts/systemd/airflow-webserver.service @@ -26,7 +26,7 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow webserver --pid /run/airflow/webserver.pid +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow webserver --pid /run/airflow/webserver.pid' Restart=on-failure RestartSec=5s PrivateTmp=true diff --git a/scripts/systemd/airflow-worker.service b/scripts/systemd/airflow-worker.service index 33776a2e82b44..610332bb9db0b 100644 --- a/scripts/systemd/airflow-worker.service +++ b/scripts/systemd/airflow-worker.service @@ -26,7 +26,7 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow celery worker +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow celery worker' Restart=on-failure RestartSec=10s