Skip to content

Commit

Permalink
Add KubernetesPodOperator & DockerOperator management (#133)
Browse files Browse the repository at this point in the history
This pull request creates two operators, `DbtKubernetesBaseOperator` and
`DbtDockerBaseOperator`, to clone the logic in the `DbtBaseOperator`
with the same subclasses (`DbtLSOperator`, `DbtSeedOperator`,
`DbtRunOperator`, ...) but to use a `KubernetesPodOperator` or a
`DockerOperator`.

I'm trying to meet a community need (see
#128 or
#97).

My PR is not perfect at all I think, but I will open it to start
discussions and make it evolve to improve it as these discussions go on.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: dimerman <danielryan2430@gmail.com>
  • Loading branch information
3 people authored Mar 20, 2023
1 parent e8dfbfd commit af87315
Show file tree
Hide file tree
Showing 62 changed files with 3,086 additions and 734 deletions.
Empty file added cosmos/core/__init__.py
Empty file.
Empty file added cosmos/core/graph/__init__.py
Empty file.
53 changes: 39 additions & 14 deletions cosmos/providers/dbt/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,53 @@
dbt support for Airflow. Contains dags, task groups, and operators.
"""

from cosmos.providers.dbt.core.operators.docker import (
DbtLSDockerOperator,
DbtRunDockerOperator,
DbtRunOperationDockerOperator,
DbtSeedDockerOperator,
DbtTestDockerOperator,
)
from cosmos.providers.dbt.core.operators.kubernetes import (
DbtLSKubernetesOperator,
DbtRunKubernetesOperator,
DbtRunOperationKubernetesOperator,
DbtSeedKubernetesOperator,
DbtTestKubernetesOperator,
)

# re-export the operators
from .core.operators import (
DbtLSOperator,
DbtRunOperationOperator,
DbtRunOperator,
DbtSeedOperator,
DbtTestOperator,
from cosmos.providers.dbt.core.operators.local import (
DbtLSLocalOperator,
DbtRunLocalOperator,
DbtRunOperationLocalOperator,
DbtSeedLocalOperator,
DbtTestLocalOperator,
)

# re-export user facing utilities
from .core.utils.data_aware_scheduling import get_dbt_dataset
from cosmos.providers.dbt.core.utils.data_aware_scheduling import get_dbt_dataset

# re-export the dag and task group
from .dag import DbtDag
from .task_group import DbtTaskGroup
from cosmos.providers.dbt.dag import DbtDag
from cosmos.providers.dbt.task_group import DbtTaskGroup

__all__ = [
DbtLSOperator,
DbtRunOperationOperator,
DbtRunOperator,
DbtSeedOperator,
DbtTestOperator,
DbtLSLocalOperator,
DbtRunOperationLocalOperator,
DbtRunLocalOperator,
DbtSeedLocalOperator,
DbtTestLocalOperator,
DbtLSDockerOperator,
DbtRunOperationDockerOperator,
DbtRunDockerOperator,
DbtSeedDockerOperator,
DbtTestDockerOperator,
DbtLSKubernetesOperator,
DbtRunOperationKubernetesOperator,
DbtRunKubernetesOperator,
DbtSeedKubernetesOperator,
DbtTestKubernetesOperator,
get_dbt_dataset,
DbtDag,
DbtTaskGroup,
Expand Down
Loading

0 comments on commit af87315

Please sign in to comment.