diff --git a/airflow-core/docs/howto/run-with-systemd.rst b/airflow-core/docs/howto/run-with-systemd.rst index 92dafd5760cd4..1006ce4a06af6 100644 --- a/airflow-core/docs/howto/run-with-systemd.rst +++ b/airflow-core/docs/howto/run-with-systemd.rst @@ -27,6 +27,12 @@ In the ``scripts/systemd`` directory, you can find unit files that have been tested on Redhat based systems. These files can be used as-is by copying them over to ``/usr/lib/systemd/system``. +You can find the latest systemd unit files on GitHub: +https://github.com/apache/airflow/tree/main/scripts/systemd + +Assumptions +----------- + The following **assumptions** have been made while creating these unit files: #. Airflow runs as the following ``user:group`` ``airflow:airflow``. @@ -34,7 +40,32 @@ The following **assumptions** have been made while creating these unit files: If this is not the case, appropriate changes will need to be made. +Environment Configuration +------------------------- + Please **note** that environment configuration is picked up from ``/etc/sysconfig/airflow``. An example file is supplied within ``scripts/systemd``. You can also define configuration at :envvar:`AIRFLOW_HOME` or :envvar:`AIRFLOW_CONFIG`. + +Using Virtual Environments +-------------------------- + +.. note:: + If Airflow is installed inside a virtual environment (e.g. ``venv`` or ``conda``), you must update the ``ExecStart`` line in each systemd unit file to activate the virtualenv first. + + Example: + + .. code-block:: ini + + ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow scheduler' + + Replace ``/home/airflow/airflow_venv/`` with the path to your virtual environment. + +New Airflow 3.0 Services +------------------------ + +Since Apache Airflow 3.0, additional components have been split out into separate services. The following new unit files are available: + +- ``airflow-triggerer.service`` for deferrable task triggering +- ``airflow-api.service`` for the standalone REST API server diff --git a/scripts/systemd/airflow-api.service b/scripts/systemd/airflow-api.service new file mode 100644 index 0000000000000..6fc5ceb3ded11 --- /dev/null +++ b/scripts/systemd/airflow-api.service @@ -0,0 +1,31 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[Unit] +Description=Airflow API Server +After=network.target + +[Service] +Environment=AIRFLOW_HOME=/home/airflow/airflow_home +User=airflow +Group=airflow +Restart=always +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow api' + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/airflow-flower.service b/scripts/systemd/airflow-flower.service index 67946630e0fb3..4056dbf98a0e5 100644 --- a/scripts/systemd/airflow-flower.service +++ b/scripts/systemd/airflow-flower.service @@ -27,6 +27,7 @@ User=airflow Group=airflow Type=simple ExecStart=/bin/airflow celery flower +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow celery flower' Restart=on-failure RestartSec=5s diff --git a/scripts/systemd/airflow-kerberos.service b/scripts/systemd/airflow-kerberos.service index edb21dd04b938..549e3550536da 100644 --- a/scripts/systemd/airflow-kerberos.service +++ b/scripts/systemd/airflow-kerberos.service @@ -26,7 +26,7 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow kerberos +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && kerberos' Restart=on-failure RestartSec=5s diff --git a/scripts/systemd/airflow-scheduler.service b/scripts/systemd/airflow-scheduler.service index 371cdc6bf840d..ada93e10125c0 100644 --- a/scripts/systemd/airflow-scheduler.service +++ b/scripts/systemd/airflow-scheduler.service @@ -26,7 +26,7 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow scheduler +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow scheduler' Restart=always RestartSec=5s diff --git a/scripts/systemd/airflow-triggerer.service b/scripts/systemd/airflow-triggerer.service new file mode 100644 index 0000000000000..f7345aa2bea16 --- /dev/null +++ b/scripts/systemd/airflow-triggerer.service @@ -0,0 +1,31 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[Unit] +Description=Airflow Triggerer +After=network.target + +[Service] +Environment=AIRFLOW_HOME=/home/airflow/airflow_home +User=airflow +Group=airflow +Restart=always +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow triggerer' + +[Install] +WantedBy=multi-user.target diff --git a/scripts/systemd/airflow-webserver.service b/scripts/systemd/airflow-webserver.service index c27022a26895d..eddeeb43bc5f0 100644 --- a/scripts/systemd/airflow-webserver.service +++ b/scripts/systemd/airflow-webserver.service @@ -26,7 +26,7 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow webserver --pid /run/airflow/webserver.pid +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow webserver --pid /run/airflow/webserver.pid' Restart=on-failure RestartSec=5s PrivateTmp=true diff --git a/scripts/systemd/airflow-worker.service b/scripts/systemd/airflow-worker.service index 33776a2e82b44..610332bb9db0b 100644 --- a/scripts/systemd/airflow-worker.service +++ b/scripts/systemd/airflow-worker.service @@ -26,7 +26,7 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow celery worker +ExecStart=/bin/bash -c 'source /home/airflow/airflow_venv/bin/activate && airflow celery worker' Restart=on-failure RestartSec=10s