Skip to content

Commit

Permalink
Update README to be specific to dbt (#258)
Browse files Browse the repository at this point in the history
  • Loading branch information
jlaneve authored Apr 30, 2023
1 parent 7df2b1d commit 978f873
Show file tree
Hide file tree
Showing 4 changed files with 100 additions and 37 deletions.
55 changes: 20 additions & 35 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
.. image:: https://github.com/astronomer/astronomer-cosmos/raw/main/docs/_static/banner.png
:align: center

.. |fury| image:: https://badge.fury.io/py/astronomer-cosmos.svg
:target: https://badge.fury.io/py/astronomer-cosmos

Expand All @@ -15,48 +12,32 @@
:alt: pre-commit.ci status


Astronomer Cosmos |fury| |ossrank| |downloads| |pre-commit|

.. image:: ./docs/cosmos-logo.svg


===========================================================

A framework for dynamically generating `Apache Airflow <https://airflow.apache.org/>`_ DAGs from other tools and frameworks. Develop your workflow in your tool of choice and render it in Airflow as a DAG or Task Group!
|fury| |ossrank| |downloads| |pre-commit|

Current support for:
- dbt
Run your dbt Core projects as `Apache Airflow <https://airflow.apache.org/>`_ DAGs and Task Groups with a few lines of code. Benefits include:

Coming soon:
- Jupyter
- Hex
- And more...open an issue if you have a request!
- Run dbt projects against Airflow connections instead of dbt profiles
- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow
- Run tests immediately after a model is done to catch issues early
- Utilize Airflow's data-aware scheduling to run models immediately after upstream ingestion
- Turn each dbt model into a task/task group complete with retries, alerting, etc.

Quickstart
__________

Check out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/#quickstart>`_.


Example Usage (dbt)
Example Usage
___________________

Cosmos lets you render dbt projects as Airflow DAGs and Task Groups. To render a DAG, import ``DbtDag`` and point it to your dbt project.

.. code-block:: python
from pendulum import datetime
from airflow import DAG
from cosmos.providers.dbt.dag import DbtDag
# dag for the project jaffle_shop
jaffle_shop = DbtDag(
dbt_project_name="jaffle_shop",
conn_id="airflow_db",
dbt_args={
"schema": "public",
},
dag_id="jaffle_shop",
start_date=datetime(2022, 11, 27),
)
Simiarly, you can render an Airflow TaskGroups using the ``DbtTaskGroup`` class. Here's an example with the jaffle_shop project:
You can render an Airflow Task Group using the ``DbtTaskGroup`` class. Here's an example with the jaffle_shop project:

.. code-block:: python
Expand All @@ -73,21 +54,25 @@ Simiarly, you can render an Airflow TaskGroups using the ``DbtTaskGroup`` class.
schedule="@daily",
):
e1 = EmptyOperator(task_id="ingestion_workflow")
e1 = EmptyOperator(task_id="pre_dbt")
dbt_tg = DbtTaskGroup(
group_id="dbt_tg",
dbt_project_name="jaffle_shop",
conn_id="airflow_db",
dbt_args={
"schema": "public",
},
)
e2 = EmptyOperator(task_id="some_extraction")
e2 = EmptyOperator(task_id="post_dbt")
e1 >> dbt_tg >> e2
This will generate an Airflow Task Group that looks like this:

.. image:: ./docs/jaffle_shop_task_group.png


Changelog
_________

Expand Down
6 changes: 4 additions & 2 deletions cosmos/providers/dbt/core/operators/lazy_load.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
class MissingPackage:
def __init__(self, module_name, optional_dependency_name):
def MissingPackage(module_name, optional_dependency_name):
def raise_error(**kwargs):
raise RuntimeError(
f"Error loading the module {module_name},"
f" please make sure the right optional dependencies are installed."
f" try - pip install astronomer-cosmos[...,{optional_dependency_name}]"
)

return raise_error
76 changes: 76 additions & 0 deletions docs/cosmos-logo.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/jaffle_shop_task_group.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 978f873

Please sign in to comment.