Skip to content

Commit

Permalink
define dag in the context of airflow (apache#46875)
Browse files Browse the repository at this point in the history
* define dag in the context of airflow

* clean up and make common docs reusable

* fix failing check
  • Loading branch information
RNHTTR authored Feb 27, 2025
1 parent 5141554 commit 74206fc
Show file tree
Hide file tree
Showing 3 changed files with 49 additions and 1 deletion.
7 changes: 6 additions & 1 deletion docs/apache-airflow/core-concepts/dags.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@
DAGs
====

A *DAG* (Directed Acyclic Graph) is the core concept of Airflow, collecting :doc:`tasks` together, organized with dependencies and relationships to say how they should run.
.. include:: ../../exts/includes/dag-definition.rst
:start-after: .. dag-definition-start
:end-before: .. dag-definition-end

Here's a basic example DAG:

Expand All @@ -30,6 +32,9 @@ It defines four Tasks - A, B, C, and D - and dictates the order in which they ha

The DAG itself doesn't care about *what* is happening inside the tasks; it is merely concerned with *how* to execute them - the order to run them in, how many times to retry them, if they have timeouts, and so on.

.. include:: ../../exts/includes/dag-definition.rst
:start-after: .. dag-etymology-start
:end-before: .. dag-etymology-end

Declaring a DAG
---------------
Expand Down
11 changes: 11 additions & 0 deletions docs/apache-airflow/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@ code" serves several purposes:
- **Extensible**: The Airflow® framework contains operators to connect with numerous technologies. All Airflow components are extensible to easily adjust to your environment.
- **Flexible**: Workflow parameterization is built-in leveraging the `Jinja <https://jinja.palletsprojects.com>`_ templating engine.

DAGs
-----------------------------------------

.. include:: ../exts/includes/dag-definition.rst
:start-after: .. dag-definition-start
:end-before: .. dag-definition-end

Take a look at the following snippet of code:

.. code-block:: python
Expand Down Expand Up @@ -78,6 +85,10 @@ seen running over time:
Each column represents one DAG run. These are two of the most used views in Airflow, but there are several
other views which allow you to deep dive into the state of your workflows.

.. include:: ../exts/includes/dag-definition.rst
:start-after: .. dag-etymology-start
:end-before: .. dag-etymology-end


Why Airflow®?
=========================================
Expand Down
32 changes: 32 additions & 0 deletions docs/exts/includes/dag-definition.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. dag-definition-start
A DAG is a model that encapsulates everything needed to execute a workflow. Some DAG attributes include the following:

* **Schedule**: When the workflow should run.
* **Tasks**: :doc:`tasks </core-concepts/tasks>` are discrete units of work that are run on workers.
* **Task Dependencies**: The order and conditions under which :doc:`tasks </core-concepts/tasks>` execute.
* **Callbacks**: Actions to take when the entire workflow completes.
* **Additional Parameters**: And many other operational details.
.. dag-definition-end
.. dag-etymology-start
.. note::

The term "DAG" comes from the mathematical concept "directed acyclic graph", but the meaning in Airflow has evolved well beyond just the literal data structure associated with the mathematical DAG concept.
.. dag-etymology-end

0 comments on commit 74206fc

Please sign in to comment.