Add dbt profile config variables to mapped profile #794

ykuc · 2024-01-11T02:31:49Z

Description

Related Issue(s)

Breaking Change?

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

netlify · 2024-01-11T02:31:56Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`434b639`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/65a30ef026f275000887c211

ykuc · 2024-01-11T02:38:21Z

@jbandoro Hi! could you check it?)

ykuc · 2024-01-11T14:18:56Z

old pr version

jbandoro

Thanks @ykuc this is looking good! Main comment below related to the breaking change that was introduced since cosmos v1.3.0 was released.

cosmos/profiles/base.py

dev/dags/basic_cosmos_dag.py

docs/templates/index.rst.jinja2

tests/profiles/test_base_profile.py

…into add-configs-in-profile-mapping

tests/profiles/test_base_profile.py

dev/dags/basic_cosmos_dag.py

codecov · 2024-01-12T22:06:23Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (1b7f4a0) 94.62% compared to head (434b639) 94.68%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #794      +/-   ##
==========================================
+ Coverage   94.62%   94.68%   +0.06%     
==========================================
  Files          55       55              
  Lines        2419     2447      +28     
==========================================
+ Hits         2289     2317      +28     
  Misses        130      130

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jbandoro

Thanks @ykuc for your contribution and work to get this ready, it looks good!

One comment below on code coverage.

cosmos/profiles/base.py

tatiana

This looks great, thanks for the great work @ykuc and @jbandoro !

@dwreeves

Features * Add dbt docs natively in Airflow via plugin by @dwreeves in #737 * Add support for ``InvocationMode.DBT_RUNNER`` for local execution mode by @jbandoro in #850 * Support partial parsing to render DAGs faster when using ``ExecutionMode.LOCAL``, ``ExecutionMode.VIRTUALENV`` and ``LoadMode.DBT_LS`` by @dwreeves in #800 * Add Azure Container Instance as Execution Mode by @danielvdende in #771 * Add dbt build operators by @dylanharper-qz in #795 * Add dbt profile config variables to mapped profile by @ykuc in #794 * Add more template fields to ``DbtBaseOperator`` by @dwreeves in #786 Bug fixes * Make ``PostgresUserPasswordProfileMapping`` schema argument optional by @FouziaTariq in #683 * Fix ``folder_dir`` not showing on logs for ``DbtDocsS3LocalOperator`` by @PrimOox in #856 * Improve ``dbt ls`` parsing resilience to missing tags/config by @tatiana in #859 * Fix ``operator_args`` modified in place in Airflow converter by @jbandoro in #835 * Fix Docker and Kubernetes operators execute method resolution by @jbandoro in #849 Docs * Fix docs homepage link by @jlaneve in #860 * Fix docs ``ExecutionConfig.dbt_project_path`` by @jbandoro in #847 * Fix typo in MWAA getting started guide by @jlaneve in #846 Others * Add performance integration tests by @jlaneve in #827 * Add ``connect_retries`` to databricks profile to fix expensive integration failures by @jbandoro in #826 * Add import sorting (isort) to Cosmos by @jbandoro in #866 * Add Python 3.11 to CI/tests by @tatiana and @jbandoro in #821, #824 and #825 * Fix failing ``test_created_pod`` for ``apache-airflow-providers-cncf-kubernetes`` after v8.0.0 update by @jbandoro in #854 * Extend ``DatabricksTokenProfileMapping`` test to include session properties by @tatiana in #858 * Fix broken integration test uncovered from Pytest 8.0 update by @jbandoro in #845 * Pre-commit hook updates in #834, #843 and #852

initially designated Pydantic as an optional dependency. However, a subsequent pull request, #794, introduced an implementation that requires Pydantic to be a mandatory dependency for astronomer-cosmos. This pull request addresses the bug encountered while running airflow db init, as described in issue #936, by enforcing Pydantic as a required dependency. Furthermore, it's worth noting that Pydantic version 2 has known issues with Airflow versions prior to 2.7.0. Given that the minimum Airflow version supported by astronomer-cosmos is 2.3, we have restricted the Pydantic version to be less than 2. Additionally, a comment has been added indicating that we may remove this upper bound once Airflow 2.7 becomes the minimum required version for astronomer-cosmos. closes: #936

The pull request #736 initially designated Pydantic as an optional dependency. However, a subsequent pull request, #794 as an [import](https://github.com/astronomer/astronomer-cosmos/pull/794/files#diff-bd3fa47d7a9b96d7bb365f3ba3b60eaf0b20e06a48b28814cf6f6e6fb64d4da6R12), introduced an implementation that requires Pydantic to be a mandatory dependency for astronomer-cosmos. This pull request addresses the bug encountered while running `airflow db init`, as described in #936, by enforcing Pydantic as a required dependency. Closes: #936

@dwreeves

Features * Add dbt docs natively in Airflow via plugin by @dwreeves in #737 * Add support for ``InvocationMode.DBT_RUNNER`` for local execution mode by @jbandoro in #850 * Support partial parsing to render DAGs faster when using ``ExecutionMode.LOCAL``, ``ExecutionMode.VIRTUALENV`` and ``LoadMode.DBT_LS`` by @dwreeves in #800 * Improve performance by 22-35% or more by caching partial parse artefact by @tatiana in #904 * Add Azure Container Instance as Execution Mode by @danielvdende in #771 * Add dbt build operators by @dylanharper-qz in #795 * Add dbt profile config variables to mapped profile by @ykuc in #794 * Add more template fields to ``DbtBaseOperator`` by @dwreeves in #786 * Add ``pip_install_options`` argument to operators by @octiva in #808 Bug fixes * Make ``PostgresUserPasswordProfileMapping`` schema argument optional by @FouziaTariq in #683 * Fix ``folder_dir`` not showing on logs for ``DbtDocsS3LocalOperator`` by @PrimOox in #856 * Improve ``dbt ls`` parsing resilience to missing tags/config by @tatiana in #859 * Fix ``operator_args`` modified in place in Airflow converter by @jbandoro in #835 * Fix Docker and Kubernetes operators execute method resolution by @jbandoro in #849 * Fix ``TrinoBaseProfileMapping`` required parameter for non method authentication by @AlexandrKhabarov in #921 * Fix global flags for lists by @ms32035 in #863 * Fix ``GoogleCloudServiceAccountDictProfileMapping`` when getting values from the Airflow connection ``extra__`` keys by @glebkrapivin in #923 * Fix using the dag as a keyword argument as ``specific_args_keys`` in DbtTaskGroup by @tboutaour in #916 * Fix ACI integration (``DbtAzureContainerInstanceBaseOperator``) by @danielvdende in #872 * Fix setting dbt project dir to the tmp dir by @dwreeves in #873 * Fix dbt docs operator to not use ``graph.gpickle`` file when ``--no-write-json`` is passed by @dwreeves in #883 * Make Pydantic a required dependency by @pankajkoti in #939 * Gracefully error if users try to ``emit_datasets`` with ``Airflow 2.9.0`` or ``2.9.1`` by @tatiana in #948 * Fix parsing tests that have no parents in #933 by @jlaneve * Correct ``root_path`` in partial parse cache by @pankajkoti in #950 Docs * Fix docs homepage link by @jlaneve in #860 * Fix docs ``ExecutionConfig.dbt_project_path`` by @jbandoro in #847 * Fix typo in MWAA getting started guide by @jlaneve in #846 * Fix typo related to exporting docs to GCS by @tboutaour in #922 * Improve partial parsing docs by @tatiana in #898 * Improve docs for datasets for airflow >= 2.4 by @SiddiqueAhmad in #879 * Improve test behaviour docs to highlight ``warning`` feature in the ``virtualenv`` mode by @mc51 in #910 * Fix docs typo by @SiddiqueAhmad in #917 * Improve Astro docs by @RNHTTR in #951 Others * Add performance integration tests by @jlaneve in #827 * Enable ``append_env`` in ``operator_args`` by default by @tatiana in #899 * Change default ``append_env`` behaviour depending on Cosmos ``ExecutionMode`` by @pankajkoti and @pankajastro in #954 * Expose the ``dbt`` graph in the ``DbtToAirflowConverter`` class by @tommyjxl in #886 * Improve dbt docs plugin rendering padding by @dwreeves in #876 * Add ``connect_retries`` to databricks profile to fix expensive integration failures by @jbandoro in #826 * Add import sorting (isort) to Cosmos by @jbandoro in #866 * Add Python 3.11 to CI/tests by @tatiana and @jbandoro in #821, #824 and #825 * Fix failing ``test_created_pod`` for ``apache-airflow-providers-cncf-kubernetes`` after v8.0.0 update by @jbandoro in #854 * Extend ``DatabricksTokenProfileMapping`` test to include session properties by @tatiana in #858 * Fix broken integration test uncovered from Pytest 8.0 update by @jbandoro in #845 * Add Apache Airflow 2.9 to the test matrix by @tatiana in #940 * Replace deprecated ``DummyOperator`` by ``EmptyOperator`` if Airflow >=2.4.0 by @tatiana in #900 * Improve logs to troubleshoot issue in 1.4.0a2 with astro-cli by @tatiana in #947 * Fix issue when publishing a new release to PyPI by @tatiana in #946 * Pre-commit hook updates in #820, #834, #843 and #852, #890, #896, #901, #905, #908, #919, #931, #941

When using `ProfileMapping` classes, allow users to define values of ``profiles.yml`` that are not specific to a particular data platform. For more information, https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml. Example of usage: ``` from cosmos.profiles import SnowflakeUserPasswordProfileMapping, DbtProfileConfigVars profile_config = ProfileConfig( profile_name="my_profile_name", target_name="my_target_name", profile_mapping=SnowflakeUserPasswordProfileMapping( conn_id="my_snowflake_conn_id", profile_args={ "database": "my_snowflake_database", "schema": "my_snowflake_schema", }, dbt_config_vars=DbtProfileConfigVars( send_anonymous_usage_stats=False, partial_parse=True, use_experimental_parse=True, static_parser=True, printer_width=120, write_json=True, warn_error=True, warn_error_options={"include": "all"}, log_format='text', debug=True, version_check=True, ), ), ) ```

@dwreeves

Features * Add dbt docs natively in Airflow via plugin by @dwreeves in astronomer#737 * Add support for ``InvocationMode.DBT_RUNNER`` for local execution mode by @jbandoro in astronomer#850 * Support partial parsing to render DAGs faster when using ``ExecutionMode.LOCAL``, ``ExecutionMode.VIRTUALENV`` and ``LoadMode.DBT_LS`` by @dwreeves in astronomer#800 * Add Azure Container Instance as Execution Mode by @danielvdende in astronomer#771 * Add dbt build operators by @dylanharper-qz in astronomer#795 * Add dbt profile config variables to mapped profile by @ykuc in astronomer#794 * Add more template fields to ``DbtBaseOperator`` by @dwreeves in astronomer#786 Bug fixes * Make ``PostgresUserPasswordProfileMapping`` schema argument optional by @FouziaTariq in astronomer#683 * Fix ``folder_dir`` not showing on logs for ``DbtDocsS3LocalOperator`` by @PrimOox in astronomer#856 * Improve ``dbt ls`` parsing resilience to missing tags/config by @tatiana in astronomer#859 * Fix ``operator_args`` modified in place in Airflow converter by @jbandoro in astronomer#835 * Fix Docker and Kubernetes operators execute method resolution by @jbandoro in astronomer#849 Docs * Fix docs homepage link by @jlaneve in astronomer#860 * Fix docs ``ExecutionConfig.dbt_project_path`` by @jbandoro in astronomer#847 * Fix typo in MWAA getting started guide by @jlaneve in astronomer#846 Others * Add performance integration tests by @jlaneve in astronomer#827 * Add ``connect_retries`` to databricks profile to fix expensive integration failures by @jbandoro in astronomer#826 * Add import sorting (isort) to Cosmos by @jbandoro in astronomer#866 * Add Python 3.11 to CI/tests by @tatiana and @jbandoro in astronomer#821, astronomer#824 and astronomer#825 * Fix failing ``test_created_pod`` for ``apache-airflow-providers-cncf-kubernetes`` after v8.0.0 update by @jbandoro in astronomer#854 * Extend ``DatabricksTokenProfileMapping`` test to include session properties by @tatiana in astronomer#858 * Fix broken integration test uncovered from Pytest 8.0 update by @jbandoro in astronomer#845 * Pre-commit hook updates in astronomer#834, astronomer#843 and astronomer#852

The pull request astronomer#736 initially designated Pydantic as an optional dependency. However, a subsequent pull request, astronomer#794 as an [import](https://github.com/astronomer/astronomer-cosmos/pull/794/files#diff-bd3fa47d7a9b96d7bb365f3ba3b60eaf0b20e06a48b28814cf6f6e6fb64d4da6R12), introduced an implementation that requires Pydantic to be a mandatory dependency for astronomer-cosmos. This pull request addresses the bug encountered while running `airflow db init`, as described in astronomer#936, by enforcing Pydantic as a required dependency. Closes: astronomer#936

@dwreeves

Features * Add dbt docs natively in Airflow via plugin by @dwreeves in astronomer#737 * Add support for ``InvocationMode.DBT_RUNNER`` for local execution mode by @jbandoro in astronomer#850 * Support partial parsing to render DAGs faster when using ``ExecutionMode.LOCAL``, ``ExecutionMode.VIRTUALENV`` and ``LoadMode.DBT_LS`` by @dwreeves in astronomer#800 * Improve performance by 22-35% or more by caching partial parse artefact by @tatiana in astronomer#904 * Add Azure Container Instance as Execution Mode by @danielvdende in astronomer#771 * Add dbt build operators by @dylanharper-qz in astronomer#795 * Add dbt profile config variables to mapped profile by @ykuc in astronomer#794 * Add more template fields to ``DbtBaseOperator`` by @dwreeves in astronomer#786 * Add ``pip_install_options`` argument to operators by @octiva in astronomer#808 Bug fixes * Make ``PostgresUserPasswordProfileMapping`` schema argument optional by @FouziaTariq in astronomer#683 * Fix ``folder_dir`` not showing on logs for ``DbtDocsS3LocalOperator`` by @PrimOox in astronomer#856 * Improve ``dbt ls`` parsing resilience to missing tags/config by @tatiana in astronomer#859 * Fix ``operator_args`` modified in place in Airflow converter by @jbandoro in astronomer#835 * Fix Docker and Kubernetes operators execute method resolution by @jbandoro in astronomer#849 * Fix ``TrinoBaseProfileMapping`` required parameter for non method authentication by @AlexandrKhabarov in astronomer#921 * Fix global flags for lists by @ms32035 in astronomer#863 * Fix ``GoogleCloudServiceAccountDictProfileMapping`` when getting values from the Airflow connection ``extra__`` keys by @glebkrapivin in astronomer#923 * Fix using the dag as a keyword argument as ``specific_args_keys`` in DbtTaskGroup by @tboutaour in astronomer#916 * Fix ACI integration (``DbtAzureContainerInstanceBaseOperator``) by @danielvdende in astronomer#872 * Fix setting dbt project dir to the tmp dir by @dwreeves in astronomer#873 * Fix dbt docs operator to not use ``graph.gpickle`` file when ``--no-write-json`` is passed by @dwreeves in astronomer#883 * Make Pydantic a required dependency by @pankajkoti in astronomer#939 * Gracefully error if users try to ``emit_datasets`` with ``Airflow 2.9.0`` or ``2.9.1`` by @tatiana in astronomer#948 * Fix parsing tests that have no parents in astronomer#933 by @jlaneve * Correct ``root_path`` in partial parse cache by @pankajkoti in astronomer#950 Docs * Fix docs homepage link by @jlaneve in astronomer#860 * Fix docs ``ExecutionConfig.dbt_project_path`` by @jbandoro in astronomer#847 * Fix typo in MWAA getting started guide by @jlaneve in astronomer#846 * Fix typo related to exporting docs to GCS by @tboutaour in astronomer#922 * Improve partial parsing docs by @tatiana in astronomer#898 * Improve docs for datasets for airflow >= 2.4 by @SiddiqueAhmad in astronomer#879 * Improve test behaviour docs to highlight ``warning`` feature in the ``virtualenv`` mode by @mc51 in astronomer#910 * Fix docs typo by @SiddiqueAhmad in astronomer#917 * Improve Astro docs by @RNHTTR in astronomer#951 Others * Add performance integration tests by @jlaneve in astronomer#827 * Enable ``append_env`` in ``operator_args`` by default by @tatiana in astronomer#899 * Change default ``append_env`` behaviour depending on Cosmos ``ExecutionMode`` by @pankajkoti and @pankajastro in astronomer#954 * Expose the ``dbt`` graph in the ``DbtToAirflowConverter`` class by @tommyjxl in astronomer#886 * Improve dbt docs plugin rendering padding by @dwreeves in astronomer#876 * Add ``connect_retries`` to databricks profile to fix expensive integration failures by @jbandoro in astronomer#826 * Add import sorting (isort) to Cosmos by @jbandoro in astronomer#866 * Add Python 3.11 to CI/tests by @tatiana and @jbandoro in astronomer#821, astronomer#824 and astronomer#825 * Fix failing ``test_created_pod`` for ``apache-airflow-providers-cncf-kubernetes`` after v8.0.0 update by @jbandoro in astronomer#854 * Extend ``DatabricksTokenProfileMapping`` test to include session properties by @tatiana in astronomer#858 * Fix broken integration test uncovered from Pytest 8.0 update by @jbandoro in astronomer#845 * Add Apache Airflow 2.9 to the test matrix by @tatiana in astronomer#940 * Replace deprecated ``DummyOperator`` by ``EmptyOperator`` if Airflow >=2.4.0 by @tatiana in astronomer#900 * Improve logs to troubleshoot issue in 1.4.0a2 with astro-cli by @tatiana in astronomer#947 * Fix issue when publishing a new release to PyPI by @tatiana in astronomer#946 * Pre-commit hook updates in astronomer#820, astronomer#834, astronomer#843 and astronomer#852, astronomer#890, astronomer#896, astronomer#901, astronomer#905, astronomer#908, astronomer#919, astronomer#931, astronomer#941

add configs in profile

f68a7b2

ykuc requested a review from a team as a code owner January 11, 2024 02:31

ykuc requested a review from a team January 11, 2024 02:31

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jan 11, 2024

ykuc had a problem deploying to external January 11, 2024 02:31 — with GitHub Actions Error

ykuc marked this pull request as draft January 11, 2024 02:32

Merge branch 'main' into add-configs-in-profile-mapping

405930b

ykuc had a problem deploying to external January 11, 2024 02:36 — with GitHub Actions Error

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

89c8a0a

pre-commit-ci bot had a problem deploying to external January 11, 2024 02:36 Error

ykuc marked this pull request as ready for review January 11, 2024 02:37

dosubot bot added area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc dbt:docs Primarily related to dbt docs command or functionality labels Jan 11, 2024

fix merge conflict

0569e69

ykuc had a problem deploying to external January 11, 2024 14:13 — with GitHub Actions Error

ykuc mentioned this pull request Jan 11, 2024

add-configs-in-profile #769

Closed

2 tasks

jbandoro reviewed Jan 11, 2024

View reviewed changes

rename DbtProfileConfigVars

eb57616

ykuc had a problem deploying to external January 11, 2024 20:38 — with GitHub Actions Error

recovery disable_event_tracking

107cbd8

ykuc had a problem deploying to external January 11, 2024 21:15 — with GitHub Actions Error

fix self.dbt_config_vars

78b6128

ykuc had a problem deploying to external January 11, 2024 21:28 — with GitHub Actions Error

fix test

1527ffc

ykuc had a problem deploying to external January 12, 2024 21:03 — with GitHub Actions Error

ykuc added 2 commits January 12, 2024 22:07

fix test

483ffa0

Merge remote-tracking branch 'origin/add-configs-in-profile-mapping' …

e758279

…into add-configs-in-profile-mapping

ykuc had a problem deploying to external January 12, 2024 21:07 — with GitHub Actions Error

jbandoro reviewed Jan 12, 2024

View reviewed changes

tests/profiles/test_base_profile.py Show resolved Hide resolved

dev/dags/basic_cosmos_dag.py Outdated Show resolved Hide resolved

jbandoro changed the title ~~add configs in profile~~ Add dbt profile config variables to mapped profile Jan 12, 2024

fix dag and test

7849c55

ykuc temporarily deployed to external January 12, 2024 21:32 — with GitHub Actions Inactive

add config test

387f34c

ykuc temporarily deployed to external January 12, 2024 21:41 — with GitHub Actions Inactive

jbandoro approved these changes Jan 12, 2024

View reviewed changes

cosmos/profiles/base.py Show resolved Hide resolved

dosubot bot added the lgtm This PR has been approved by a maintainer label Jan 12, 2024

add test test_dbt_profile_config_vars_none

434b639

ykuc temporarily deployed to external January 13, 2024 22:30 — with GitHub Actions Inactive

tatiana approved these changes Jan 18, 2024

View reviewed changes

tatiana merged commit d3ea329 into astronomer:main Jan 18, 2024

tatiana mentioned this pull request Feb 27, 2024

Release 1.4.0a1 #864

Merged

tatiana mentioned this pull request May 2, 2024

Release 1.4.0 #934

Merged

pankajkoti mentioned this pull request May 3, 2024

Make Pydantic a required dependency #939

Merged

2 tasks

tatiana mentioned this pull request Jun 4, 2024

Remove dependency on Pydantic #1020

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add dbt profile config variables to mapped profile #794

Add dbt profile config variables to mapped profile #794

Uh oh!

ykuc commented Jan 11, 2024 •

edited

Loading

Uh oh!

netlify bot commented Jan 11, 2024 •

edited

Loading

Uh oh!

ykuc commented Jan 11, 2024

Uh oh!

ykuc commented Jan 11, 2024

Uh oh!

jbandoro left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 12, 2024 •

edited

Loading

Uh oh!

jbandoro left a comment

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Add dbt profile config variables to mapped profile #794

Add dbt profile config variables to mapped profile #794

Uh oh!

Conversation

ykuc commented Jan 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Breaking Change?

Checklist

Uh oh!

netlify bot commented Jan 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

ykuc commented Jan 11, 2024

Uh oh!

ykuc commented Jan 11, 2024

Uh oh!

jbandoro left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jbandoro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ykuc commented Jan 11, 2024 •

edited

Loading

netlify bot commented Jan 11, 2024 •

edited

Loading

jbandoro left a comment •

edited

Loading

codecov bot commented Jan 12, 2024 •

edited

Loading