-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor LoadMethod.LOCAL to use symlinks as opposed to copying the dbt source dir #614
Labels
area:performance
Related to performance, like memory usage, CPU usage, speed, etc
execution:local
Related to Local execution environment
good first issue
Good for newcomers
Milestone
Comments
tatiana
added
enhancement
New feature or request
good first issue
Good for newcomers
labels
Oct 19, 2023
tatiana
added
area:performance
Related to performance, like memory usage, CPU usage, speed, etc
and removed
enhancement
New feature or request
labels
Oct 19, 2023
I can work on this one, in #629 to reduce complexity I created the function: astronomer-cosmos/cosmos/dbt/graph.py Lines 56 to 61 in 9cd46d2
which could be used here in |
2 tasks
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this issue
Jul 14, 2024
astronomer#660) This PR refactors the `create_symlinks` function that was previously used in load via dbt ls so that it can be used in `DbtLocalBaseOperator.run_command` instead of copying the entire directory. Closes: astronomer#614
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area:performance
Related to performance, like memory usage, CPU usage, speed, etc
execution:local
Related to Local execution environment
good first issue
Good for newcomers
As of Cosmos 1.0-1.2, it copies the entire dbt project for every operator that runs dbt:
astronomer-cosmos/cosmos/operators/local.py
Line 198 in c82e6bc
As reported in the #airflow-dbt slack, this can lead to performance issues:
https://apache-airflow.slack.com/archives/C059CC42E9W/p1697718075031609?thread_ts=1697708239.918849&cid=C059CC42E9W
We successfully adopted a strategy of using symbolic links from a temporary folder to run
dbt ls
:astronomer-cosmos/cosmos/dbt/graph.py
Lines 167 to 171 in c82e6bc
The goal with this ticket is to adopt this same strategy when running tasks using
LoadMethod.LOCAL
.The text was updated successfully, but these errors were encountered: