Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate columns for dbt and Airflow links in the concept comparison Doc #542

Merged
merged 5 commits into from
Sep 23, 2023
Merged

Separate columns for dbt and Airflow links in the concept comparison Doc #542

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
425100d
seperate out reference column
TJaniF Sep 15, 2023
0ebec73
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] Sep 15, 2023
9cefab6
hyperlinked table
TJaniF Sep 18, 2023
3d4c4cf
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] Sep 18, 2023
b55a796
Merge branch 'main' into reference_cols
tatiana Sep 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 14 additions & 24 deletions docs/getting_started/dbt-airflow-concepts.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,18 @@ differences, they also share similar concepts.
This page aims to list some of these concepts and help those
who may be new to Airflow or dbt and are considering to use Cosmos.

.. table::
:align: left
:widths: auto

+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Airflow naming | dbt naming | Description | Differences | References |
+================+==============+=================================================================================+=============================================================================+======================================================================================+
| DAG | Workflow | Pipeline (Direct Acyclic Graph) that contains a group of steps | Airflow expects upstream tasks to have passed to run downstream tasks. | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/dags.html |
| | | | dbt can run a subset of tasks assuming upstream tasks were run. | https://docs.getdbt.com/docs/introduction |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Task | Node | Step within a pipeline (DAG or workflow) | In dbt, these are usually transformations that run on a remote database. | https://docs.getdbt.com/reference/node-selection/syntax |
| | | | In Airflow, steps can be anything, running locally in Airflow or remotely. | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/tasks.html |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Language | Language | Programming or declarative language used to define pipelines and steps. | In dbt, users write SQL, YML and Python to define the steps of a pipeline. | https://docs.getdbt.com/docs/introduction#dbt-optimizes-your-workflow |
| | | | Airflow expects steps and pipelines are written in Python. | https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Variables | Variables | Key-value configuration that can be used in steps and avoids hard-coded values | | https://docs.getdbt.com/docs/build/project-variables |
| | | | | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/variables.html |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Templating | Macros | Jinja templating used to access variables, configuration and reference steps | dbt encourages using jinja templating for control structures (if and for). | https://docs.getdbt.com/docs/build/jinja-macros |
| | | | Native in Airflow/Python, used to define variables, macros and filters. | https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Connection | Profile | Configuration to connect to databases or other services | | https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html |
| | | | | https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Providers | Adapter | Additional Python libraries that support specific databases or services | | https://airflow.apache.org/docs/apache-airflow-providers/ |
| | | | | https://docs.getdbt.com/guides/dbt-ecosystem/adapter-development/1-what-are-adapters |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
=================================================================================================== ==================================================================================================== ==================================================================================== ======================================================================================================================================================
Airflow naming dbt naming Description Differences
=================================================================================================== ==================================================================================================== ==================================================================================== ======================================================================================================================================================
`DAG <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html>`_ `Workflow <https://docs.getdbt.com/docs/introduction>`_ Pipeline (Direct Acyclic Graph) that contains a group of steps Airflow expects upstream tasks to have passed to run downstream tasks. dbt can run a subset of tasks assuming upstream tasks were run.
`Task <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html>`_ `Node <https://docs.getdbt.com/reference/node-selection/syntax>`_ Step within a pipeline (DAG or workflow) In dbt, these are usually transformations that run on a remote database. In Airflow, steps can be anything, running locally in Airflow or remotely.
`Language <https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html>`_ `Language <https://docs.getdbt.com/docs/introduction#dbt-optimizes-your-workflow>`_ Programming or declarative language used to define pipelines and steps. In dbt, users write SQL, YML and Python to define the steps of a pipeline. Airflow expects steps and pipelines are written in Python.
`Variables <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/variables.html>`_ `Variables <https://docs.getdbt.com/docs/build/project-variables>`_ Key-value configuration that can be used in steps and avoids hard-coded values
`Templating <https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html>`_ `Macros <https://docs.getdbt.com/docs/build/jinja-macros>`_ Jinja templating used to access variables, configuration and reference steps dbt encourages using jinja templating for control structures (if and for). Native in Airflow/Python, used to define variables, macros and filters.
`Connection <https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html>`_ `Profile <https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles>`_ Configuration to connect to databases or other services
`Providers <https://airflow.apache.org/docs/apache-airflow-providers/>`_ `Adapter <https://docs.getdbt.com/guides/dbt-ecosystem/adapter-development/1-what-are-adapters>`_ Additional Python libraries that support specific databases or services
=================================================================================================== ==================================================================================================== ==================================================================================== ======================================================================================================================================================