Add new parsing method `LoadMode.DBT_LS_FILE` #733

woogakoki · 2023-12-01T08:52:38Z

Description

Adds a new load method which is in between manifest and dbt_ls. Goal of this is to have a dbt ls file with the output prepared and only use the parsing logic of dbt_ls load method. This should increase performance compared to using dbt_ls.
However this method needs one output file for each taskgroup build.

Breaking Change?

No breaking changes, it just adds a new load method on top but leaves the rest as is.

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

…ead of running dbt ls localy

netlify · 2023-12-01T08:52:45Z

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	`5943619`
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/657ab7e1e103910008008a31

codecov · 2023-12-01T13:37:06Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (4369987) 93.18% compared to head (5943619) 93.23%.
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #733      +/-   ##
==========================================
+ Coverage   93.18%   93.23%   +0.04%     
==========================================
  Files          55       55              
  Lines        2451     2469      +18     
==========================================
+ Hits         2284     2302      +18     
  Misses        167      167

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tatiana

Hi @woogakoki, thanks a lot for the contribution. This looks great!

Your change will help with the performance for users who like the precision of the DBT_LS load method but don't want to have the pay the cost of running it during DAG parsing/task execution.

Three main feedback points:

Please, could you address the pre-commit errors
The coverage checks are also failing, if you could increase the test coverage by addressing the PR comments created by Codecov, it would be amazing!
Please, could you change one of the existing example DAGs in dev/dags to use this parsing method? It could perhaps be this one: https://github.com/astronomer/astronomer-cosmos/blob/main/dev/dags/user_defined_profile.py

I left some minor comments inline.

docs/configuration/parsing-methods.rst

tatiana · 2023-12-07T02:11:42Z

Hey @woogakoki, we're super close to getting this merged!
Once you address the code coverage comments, we can release it as part of a 1.3 alpha.

woogakoki · 2023-12-07T07:28:52Z

Hey @woogakoki, we're super close to getting this merged! Once you address the code coverage comments, we can release it as part of a 1.3 alpha.

Hi @tatiana, thanks for the heads up. Will try to fix it over the course of today then!

woogakoki · 2023-12-12T16:22:58Z

@woogakoki Just added some last comments before we can merge this feature. Can't wait to try this out. Do you by any chance have numbers on the performance improvement by using this method?

Unfortunatly not as I only tested it locally

woogakoki · 2023-12-13T08:24:31Z

@tatiana The Type-Check test is failing due to some issue in cosmos/operators/kubernetes.py. Is that known and can be ignored?

tatiana · 2023-12-13T13:28:58Z

@woogakoki Thanks for addressing the feedback. Yes, the issue with the type check was solved by @jbandoro in #766. Once the checks pass, I'll merge your PR!

tatiana · 2023-12-13T13:40:50Z

@woogakoki when you get a chance, please, check the failing tests:

FAILED tests/dbt/test_graph.py::test_load_via_dbt_ls_file - cosmos.dbt.graph.CosmosLoadDbtException: Unable to load dbt ls file without RenderConfig.project_path
===== 1 failed, 22 passed, 3 skipped, 330 deselected in 459.96s (0:07:39) ======

Also, once you rebase, we should see the type check passing 🤞

…into feat/add_new_loadmode

dev/dags/dbt/jaffle_shop/dbt_ls_models_staging.txt

tatiana · 2023-12-14T11:28:38Z

Amazing work, @woogakoki , thank you very much for getting this work through

woogakoki · 2023-12-14T12:19:21Z

Amazing work, @woogakoki , thank you very much for getting this work through

Thank you for the support @tatiana! Super happy it went through this year already.

@woogakoki

**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in #733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in #728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in #755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in #540, #688 and #741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in #608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in #616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in #648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in #735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in #768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in #730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in #663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in #758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in #759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in #772 **Others** * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Speed up integration tests by @jbandoro in #732 * Fix README quickstart link in by @RNHTTR in #776 * Add package location to work with hatchling 1.19.0 by @jbandoro in #761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in #766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in #757 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 and #779 * pre-commit updates in #775, #770, #762

@woogakoki

**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in astronomer#755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in astronomer#616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in astronomer#768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in astronomer#758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in astronomer#759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in astronomer#772 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Speed up integration tests by @jbandoro in astronomer#732 * Fix README quickstart link in by @RNHTTR in astronomer#776 * Add package location to work with hatchling 1.19.0 by @jbandoro in astronomer#761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in astronomer#766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 and astronomer#779 * pre-commit updates in astronomer#775, astronomer#770, astronomer#762

Adds a new load method, which is in between manifest and dbt_ls. This aims to have a dbt ls file with the output prepared and only use the parsing logic of the dbt_ls load method. This should increase performance compared to using dbt_ls. However, this method needs one output file for each taskgroup build.

@woogakoki

**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in astronomer#755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in astronomer#616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in astronomer#768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in astronomer#758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in astronomer#759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in astronomer#772 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Speed up integration tests by @jbandoro in astronomer#732 * Fix README quickstart link in by @RNHTTR in astronomer#776 * Add package location to work with hatchling 1.19.0 by @jbandoro in astronomer#761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in astronomer#766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 and astronomer#779 * pre-commit updates in astronomer#775, astronomer#770, astronomer#762

woogakoki added 4 commits November 30, 2023 08:51

adds LoadMode dbt_ls_file to render tasks using dbt ls file path inst…

da7fa77

…ead of running dbt ls localy

moves dbt_ls_path to render config adds some test and documentation

e12eb17

adds missing space in documentation

93c1557

points out the advantage of new load method more

4c1e37b

woogakoki requested a review from a team as a code owner December 1, 2023 08:52

woogakoki requested a review from a team December 1, 2023 08:52

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Dec 1, 2023

woogakoki had a problem deploying to external December 1, 2023 08:52 — with GitHub Actions Error

dosubot bot added area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:parse Primarily related to dbt parse command or functionality parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc labels Dec 1, 2023

fix exception message

20ea7b0

woogakoki temporarily deployed to external December 1, 2023 08:56 — with GitHub Actions Inactive

tatiana reviewed Dec 4, 2023

View reviewed changes

docs/configuration/parsing-methods.rst Show resolved Hide resolved

tatiana added this to the 1.3.0 milestone Dec 4, 2023

fix pre-commit

92accdb

woogakoki had a problem deploying to external December 5, 2023 14:49 — with GitHub Actions Error

fix pre-commit

9f7ac1c

woogakoki temporarily deployed to external December 5, 2023 14:52 — with GitHub Actions Inactive

tatiana changed the title ~~Feat/add new loadmode~~ Add new parsing method LoadMode.DBT_LS_FILE Dec 5, 2023

adds test coverage for missing files and paths

b8c4e83

woogakoki temporarily deployed to external December 7, 2023 07:54 — with GitHub Actions Inactive

fix formatting

39d37dd

woogakoki temporarily deployed to external December 7, 2023 07:57 — with GitHub Actions Inactive

fix render config test

70d4fd3

woogakoki had a problem deploying to external December 12, 2023 16:13 — with GitHub Actions Error

fix pre-commit

0e4b11c

woogakoki had a problem deploying to external December 12, 2023 16:20 — with GitHub Actions Error

use render_config.project_path

a42ba9c

woogakoki had a problem deploying to external December 12, 2023 16:29 — with GitHub Actions Error

fix test

a539c3a

woogakoki temporarily deployed to external December 12, 2023 16:33 — with GitHub Actions Inactive

woogakoki requested a review from tatiana December 13, 2023 07:44

tatiana approved these changes Dec 13, 2023

View reviewed changes

tatiana added the lgtm This PR has been approved by a maintainer label Dec 13, 2023

fix test_load_via_dbt_ls_file

492602d

woogakoki had a problem deploying to external December 13, 2023 15:48 — with GitHub Actions Error

Merge branch 'main' of https://github.com/astronomer/astronomer-cosmos …

9692b0c

…into feat/add_new_loadmode

woogakoki temporarily deployed to external December 13, 2023 15:48 — with GitHub Actions Inactive

tatiana reviewed Dec 13, 2023

View reviewed changes

dev/dags/dbt/jaffle_shop/dbt_ls_models_staging.txt Show resolved Hide resolved

include seeds in the example

5943619

woogakoki temporarily deployed to external December 14, 2023 08:08 — with GitHub Actions Inactive

tatiana mentioned this pull request Dec 14, 2023

Additions to docs regarding the DBT_MANIFEST load mode #757

Merged

tatiana merged commit 82e8db9 into astronomer:main Dec 14, 2023
40 checks passed

tatiana mentioned this pull request Jan 4, 2024

Release 1.3.0 #781

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new parsing method `LoadMode.DBT_LS_FILE` #733

Add new parsing method `LoadMode.DBT_LS_FILE` #733

woogakoki commented Dec 1, 2023 •

edited

Loading

netlify bot commented Dec 1, 2023 •

edited

Loading

codecov bot commented Dec 1, 2023 •

edited

Loading

tatiana left a comment

tatiana commented Dec 7, 2023

woogakoki commented Dec 7, 2023 •

edited

Loading

woogakoki commented Dec 12, 2023

woogakoki commented Dec 13, 2023

tatiana commented Dec 13, 2023

tatiana commented Dec 13, 2023 •

edited

Loading

tatiana commented Dec 14, 2023

woogakoki commented Dec 14, 2023

Add new parsing method LoadMode.DBT_LS_FILE #733

Add new parsing method LoadMode.DBT_LS_FILE #733

Conversation

woogakoki commented Dec 1, 2023 • edited Loading

Description

Breaking Change?

Checklist

netlify bot commented Dec 1, 2023 • edited Loading

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

codecov bot commented Dec 1, 2023 • edited Loading

Codecov Report

tatiana left a comment

Choose a reason for hiding this comment

tatiana commented Dec 7, 2023

woogakoki commented Dec 7, 2023 • edited Loading

woogakoki commented Dec 12, 2023

woogakoki commented Dec 13, 2023

tatiana commented Dec 13, 2023

tatiana commented Dec 13, 2023 • edited Loading

tatiana commented Dec 14, 2023

woogakoki commented Dec 14, 2023

Add new parsing method `LoadMode.DBT_LS_FILE` #733

Add new parsing method `LoadMode.DBT_LS_FILE` #733

woogakoki commented Dec 1, 2023 •

edited

Loading

netlify bot commented Dec 1, 2023 •

edited

Loading

codecov bot commented Dec 1, 2023 •

edited

Loading

woogakoki commented Dec 7, 2023 •

edited

Loading

tatiana commented Dec 13, 2023 •

edited

Loading