Tasks are running full parse each time

[DanMawdsleyBA]

I've noticed that within each job it is running a full parse each time. I upload the manifest as part of the cicd so not sure why it isn't recognizing the existing manifest file. This could significantly improve the job performance currently full parse step takes 7 seconds per task.

[dosubot]

🤖

Hello @DanMawdsleyBA,

Welcome to the Cosmos repository! I'm Dosu, a bot designed to assist you with solving bugs, answering questions, and guiding you through your contributions. While we wait for a human maintainer, I'm here to help. Let's get your issue addressed!

The issue you're experiencing is due to the way Cosmos is currently loading your dbt project. In the load method of the DbtGraph class, Cosmos checks if a manifest file is available. If it is, it loads the project from the manifest. If not, it tries to load the project using the dbt ls command or a custom parser.

However, even when the manifest is available, Cosmos doesn't use it to avoid parsing the project every time a job runs. Instead, it parses the project again to apply filters (based on self.exclude and self.select), which can be time-consuming.

To improve job performance, you can modify the load method to skip parsing if the manifest is available and no filters are applied. Here's an example of how you can do this:

def load(
    self,
    method: LoadMode = LoadMode.AUTOMATIC,
    execution_mode: ExecutionMode = ExecutionMode.LOCAL,
) -> None:
    if method == LoadMode.AUTOMATIC:
        if self.project.is_manifest_available() and not self.render_config.select and not self.render_config.exclude:
            self.load_from_dbt_manifest()
        else:
            if execution_mode == ExecutionMode.LOCAL and self.profile_config:
                try:
                    self.load_via_dbt_ls()
                except FileNotFoundError:
                    self.load_via_custom_parser()
            else:
                self.load_via_custom_parser()
    else:
        load_method[method]()

In this modified version of the load method, Cosmos will load the project from the manifest and skip parsing if the manifest is available and no filters are applied. If filters are applied, Cosmos will continue to parse the project as it currently does. This should improve job performance by avoiding unnecessary parsing.

You can find more details about the load method and its current implementation in the DbtGraph class.

Please let me know if you have any other questions or need further clarification.

Sources

• cosmos/dbt/graph.py

[DanMawdsleyBA]

I think this is due to the syslink again as it does not copy across the target path by default. We could add a step to check if the manifest file is present then syslink just the manifest file. This should mean each task does not need to be reparsed fully each time if the manifest file is already present. @tatiana any thoughts on the above

[tatiana]

@DanMawdsleyBA I agree that we should allow users to cache the manifest since it would improve the performance.

What happens if users are using LoadMode.DBT_LS and the manifest is outdated?

I believe we should symlink the manifest when users opt for LoadMode.DBT_MANIFEST and define manifest_path. Perhaps for the other load modes we shouldn't do this by default, but give users the option do to so.

[lrcuwicRSG1]

Hi @ALL, I am new to astronomer cosmos and I have the same performance issues as @DanMawdsleyBA noticed. When will it be fixed? @tatiana Milestone 1.4.0 was updated for the release date

In the meanwhile I want to fix it temporary with the codesnippet from dosubot. What is the easiest way to modiy the graph.py file and take it over to my local docker-installation based on astronomer-runtime and astro-cli? I am looking forward to your suggestions. Thanks Chris

[tatiana]

@DanMawdsleyBA @lrcuwicRSG1 Around two weeks ago, we managed to have an alpha release with #800 by @dwreeves, which seems to solve the problem. Could you try it out using 1.4.0a1? We're planning to release the stable 1.4 with this feature next week, and it would be great to confirm if/how much it helped you

[lrcuwicRSG1]

@tatiana Thanks for your reply. The issue #800 was on my watchlist ;) I will try it soon. Cheers Chris

[DanMawdsleyBA]

I am using 1.4.0a1 and wasn't able to get it to work because DBT was saying the profile and env vars have changed.

[2024-03-14, 17:04:56 GMT] {{subprocess.py:94}} INFO - �[0m17:04:56 Running with dbt=1.7.9
[2024-03-14, 17:04:57 GMT] {{subprocess.py:94}} INFO - �[0m17:04:57 Registered adapter: snowflake=1.7.2
[2024-03-14, 17:04:57 GMT] {{subprocess.py:94}} INFO - �[0m17:04:57 Unable to do partial parsing because config vars, config profile, or config target have changed
[2024-03-14, 17:04:57 GMT] {{subprocess.py:94}} INFO - �[0m17:04:57 Unable to do partial parsing because profile has changed
[2024-03-14, 17:04:57 GMT] {{subprocess.py:94}} INFO - �[0m17:04:57 Unable to do partial parsing because env vars used in profiles.yml have changed
[2024-03-14, 17:05:00 GMT] {{subprocess.py:94}} INFO - �[0m17:05:00 Found 47 models, 12 snapshots, 0 sources, 0 exposures, 0 metrics, 554 macros, 0 groups, 0 semantic models

In my cicd pipeline I use a dummy target to generate the manifest. As cosmos uses a different target and generates it own in env vars I'm not sure if it will work? (I could be wrong I also use env vars for other things as well).

May also need to write a guide on how to get it to work for others. For example the dbt version in the cicd has to be exactly the same as the airflow. Possibly the env_vars as well? And the dbt target?

Haven't had time to do a proper investigation as we're upping our airflow instance to 2.8.

It uses the cosmos managed profiles for snowflake which uses env vars.

[dwreeves]

I was able to get it to work but it's a lot of effort. Later today I'll post the full guide. I think I actually posted it elsewhere in a comment in this repo.

[tatiana]

Thank you very much for the feedback, @DanMawdsleyBA !

To confirm: you generated both manifest.json and partial_parse.msgpack in the CI. The DAG rendered fine in Airflow with Cosmos 1.4.0a1, but when you attempted to run the tasks, you got these logs, meaning that the tasks ran without benefiting from partial parsing. Is this the case?

I was just reading dbt docs, and it seems when dbt attempts to use partial parsing, it needs to ensure that the environment variables used in profiles.yml remain consistent:
https://docs.getdbt.com/reference/parsing#known-limitations

If certain inputs change between runs, dbt will trigger a full re-parse. The results will be correct, but the full re-parse may be quite slow. Today those inputs are:

• --vars
• profiles.yml content (or env_var values used within)
• dbt_project.yml content (or env_var values used within)
• installed packages
• dbt version
• certain widely-used macros (for example, builtins, overrides, or generate_x_name for database/schema/alias)

If you're triggering CI job runs, the benefits of partial parsing are not applicable to new pull requests (PR) or new branches. However, they are applied on subsequent commits to the new PR or branch.

So, for users to fully benefit from this feature (partial parsing), we'd need to have the same profiles.yml in the CI, when parsing the DAG, and when running the tasks. At this moment, this would mean not using Cosmos profile mapping and necessarily specifying ProfileConfig(profiles_yml_filepath=).

@dwreeves, should we improve the docs to make this requirement more explicit? Let me know your thoughts: #898

[dwreeves]

I do agree that specifying a profiles_yml_filepath is the best way to go for most users. It should be what the docs suggest.

It's also possible to do something like this:

1. Go to one of your DbtOperator task logs in Airflow, and copy the profile YAML you see in the logs.
   It will look something like this (example of a Snowflake profile):

   my_profile_name:
       outputs:
           airflow_target:
               account: myaccount
               database: mydatabase
               password: '{{ env_var(''COSMOS_CONN_SNOWFLAKE_PASSWORD'') }}'
               # ... etc. ...
       target: my_target_name

2. Add this file to your Airflow project somewhere, e.g. dags/dbt/profile/profiles.yml.

3. In your CICD, set the DBT_PROFILES_DIR env var to the folder you placed that file, e.g. dags/dbt/profile.

4. Add --profile [my_profile_name] --target [my_target_name] to the dbt parse command you run in your CICD (replace [my_profile_name] and [my_target_name] with the appropriate names). You must add these flags and you cannot just rely on defaults. This is because dbt partial parsing does a checksum on a subset of the flags used in the CLI, and both --profile and --target are included in the checksum. Failing to add these flags means it won't work.

   dbt deps
   dbt parse --profile [my_profile_name] --target [my_target_name]

5. You must set up your CICD to run the exact same connection you use in Cosmos, meaning that you need to set COSMOS_CONN_SNOWFLAKE_PASSWORD to be the password you use in production. Again, similar deal as above: dbt does checksums when parsing, and every call to the env_var() macro creates a side-effect where the key-value pair of the env var is eventually put into the hash.

6. Any --vars and env vars you use must also be included the exact same way you use them in Airflow. You cannot rely on DagRun-specific or DAG-specific parameterized vars / env vars, otherwise partial parsing won't work. (The 1.5.0 or later updates to partial parsing support should address this limitation, though.)