Make profiles_yml_path optional for ExecutionMode.DOCKER and KUBERNETES
#681
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…h profile_mapping and profiles_yml_filepath are defined
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
dosubot
bot
added
area:config
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #681 +/- ##
==========================================
+ Coverage 92.77% 92.88% +0.10%
==========================================
Files 55 55
Lines 2243 2248 +5
==========================================
+ Hits 2081 2088 +7
+ Misses 162 160 -2 ☔ View full report in Codecov by Sentry. |
tatiana
reviewed
Nov 16, 2023
There was a problem hiding this comment.
This is looking great, @MrBones757 , one minor comment, and we're good to merge!
tatiana
changed the title
profiles_yml_path optional for ExecutionMode.DOCKER and KUBERNETES
tatiana
approved these changes
Nov 18, 2023
There was a problem hiding this comment.
Thanks, @MrBones757 , when using ExecutionMode.KUBERNETES and LoadMode.DBT_MANIFEST, we really should not enforce the profiles_yml_path to be set.
This was referenced Nov 23, 2023
Merged
Merged
pull bot
pushed a commit
to pgoslatara/astronomer-cosmos
that referenced
this pull request
Nov 24, 2023
Bug fixes * Fix running models that use alias while supporting dbt versions by @binhnq94 in astronomer#662 * Make profiles_yml_path optional for ExecutionMode.DOCKER and KUBERNETES by @MrBones757 in astronomer#681 * Prevent overriding dbt profile fields with profile args of type or method by @jbandoro in astronomer#702 * Fix LoadMode.DBT_LS fail when dbt outputs WarnErrorOptions by @adammarples in astronomer#692 * Add support for env vars in RenderConfig for dbt ls parsing by @jbandoro in astronomer#690 * Add support for Kubernetes on_warning_callback by @david-mag in astronomer#673 * Fix ExecutionConfig.dbt_executable_path to use ``default_factory`` by @jbandoro in astronomer#678 Others * Docs fix: example DAG in the README and docs/index by @tatiana in astronomer#705 * Docs improvement: highlight DAG examples in README by @iancmoritz and @jlaneve in astronomer#695 (cherry picked from commit 2878d6a)
tatiana
added a commit
that referenced
this pull request
Dec 7, 2023
Features * Add ProfileMapping for Vertica by @perttus in #540 and #688 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648 * Add operator_args full_refresh as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735 Enhancements * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730 * Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731 * Add aws_session_token for Athena mapping by @benjamin-awd in #663 Others * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732
Merged
tatiana
added a commit
that referenced
this pull request
Dec 7, 2023
Features * Add ProfileMapping for Vertica by @perttus in #540 and #688 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648 * Add operator_args full_refresh as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735 Enhancements * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730 * Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731 * Add aws_session_token for Athena mapping by @benjamin-awd in #663 Others * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732
jbandoro
pushed a commit
that referenced
this pull request
Dec 7, 2023
**Features** * Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in #608 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in #616 * Add support to select using (some) graph operators when using `LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in #648 * Add operator_args ``full_refresh`` as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in #735 **Enhancements** * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to `dbt_packages` when `dbt_deps` is False when using `LoadMode.DBT_LS` by @DanMawdsleyBA in #730 * Support no `profile_config` for `ExecutionMode.KUBERNETES` and `ExecutionMode.DOCKER` by @MrBones757 and @tatiana in #681 and #731 * Add `aws_session_token` for Athena mapping by @benjamin-awd in #663 **Others** * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
…ERNETES` (astronomer#681) This PR moves validation of the `profiles_yml_path` to later in the dag generation process such that additional context can be gathered from `ExecutionConfig` to avoid failing unnecessarily when the file does not exist. Closes: astronomer#680 Closes: astronomer#656
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
Bug fixes * Fix running models that use alias while supporting dbt versions by @binhnq94 in astronomer#662 * Make profiles_yml_path optional for ExecutionMode.DOCKER and KUBERNETES by @MrBones757 in astronomer#681 * Prevent overriding dbt profile fields with profile args of type or method by @jbandoro in astronomer#702 * Fix LoadMode.DBT_LS fail when dbt outputs WarnErrorOptions by @adammarples in astronomer#692 * Add support for env vars in RenderConfig for dbt ls parsing by @jbandoro in astronomer#690 * Add support for Kubernetes on_warning_callback by @david-mag in astronomer#673 * Fix ExecutionConfig.dbt_executable_path to use ``default_factory`` by @jbandoro in astronomer#678 Others * Docs fix: example DAG in the README and docs/index by @tatiana in astronomer#705 * Docs improvement: highlight DAG examples in README by @iancmoritz and @jlaneve in astronomer#695 (cherry picked from commit 2878d6a)
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
**Features** * Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in astronomer#616 * Add support to select using (some) graph operators when using `LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in astronomer#728 * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648 * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to `dbt_packages` when `dbt_deps` is False when using `LoadMode.DBT_LS` by @DanMawdsleyBA in astronomer#730 * Support no `profile_config` for `ExecutionMode.KUBERNETES` and `ExecutionMode.DOCKER` by @MrBones757 and @tatiana in astronomer#681 and astronomer#731 * Add `aws_session_token` for Athena mapping by @benjamin-awd in astronomer#663 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 * Speed up integration tests by @jbandoro in astronomer#732
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR moves validation of the profiles_yml_path to later in the dag generation process such that additional context can be gathered from ExecutionConfig to avoid failing unnecessarily when the file does not exist.
Related Issue(s)
closes #680
closes #656
Breaking Change?
N/A
Checklist