Dataset only emit for specific models

[DanMawdsleyBA]

Description

We're looking into leveraging datasets but one problem we're having is the ability to control which models produce datasets. Would it be possible to implement something to be able to control which models produce datasets. Perhaps using a list or regex of models to emit datasets?

Use case/motivation

As part of dbt we use a lot of working tables were we don't want to emit datasets and want to cut down on the number of datasets as the airflow graph can get hard to use.

Related issues

No response

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!

[tatiana]

@DanMawdsleyBA I haven't tried this, but it may be possible with Cosmos 1.8., thanks to @wornjs' #1339 contribution:
https://astronomer.github.io/astronomer-cosmos/getting_started/custom-airflow-properties.html

You could try to set RenderConfig(emit_datasets=False) at a DbtDag or DbtTaskGroup level, and in the models you wanted to emit datasets, you could try to configure:

version: 2

models:
  - name: name
    description: description
    meta:
      owner: 'jaegwon.seo@toss.im'
      cosmos:
        operator_kwargs:
          emit_datasets: True

Since this parameter is set at an operator level:

astronomer-cosmos/cosmos/operators/base.py

Line 34 in 184e45f

:param emit_datasets: Enable emitting inlets and outlets during task execution

Could you please try it and let us know how it goes?

[DanMawdsleyBA]

I have had a try of this which doesn't seem to work. I also tried setting the emit datasets to false from the operator args:

        operator_args={'dbt_cmd_flags': ['--exclude', 'test_type:unit']
                       , 'full_refresh': True
                       , "emit_datasets": False}

This doesn't work and still emits the datasests. The only way I have managed to turn them off is within the render_config:

        render_config=RenderConfig(
            emit_datasets=False)

But this will still turn them off for the entire dag.

Wonder if this is because of the behavior in converter.py

        task_args = {
            **operator_args,
            "project_dir": execution_config.project_path,
            "partial_parse": project_config.partial_parse,
            "profile_config": profile_config,
            "emit_datasets": render_config.emit_datasets,
            "env": env_vars,
            "vars": dbt_vars,
        }

For the emit_datasets its only using the value defined from the render config

[tatiana]

@DanMawdsleyBA could you try out Cosmos 1.9.0a4 and see if emit_datasets gets overriden?

In https://github.com/astronomer/astronomer-cosmos/pull/1492/files#diff-1346b2a26d27ccd0db1837b746c9e188afa4dc499040c9cedf612ad6bc208459R178 I changed how we override the parameters - and this will likely work now.

[DanMawdsleyBA]

Hi @tatiana yeah it seems to work now! I'm using 1.9.0a5 . I'm also getting a lot of extra logging not sure if this expected given its an alpha release?

[2025-02-05, 15:14:31 UTC] {connection.py:413} INFO - Snowflake Connector for Python Version: 3.12.1, Python Version: 3.11.9, Platform: Linux-5.10.230-223.885.amzn2.x86_64-x86_64-with-glibc2.34
[2025-02-05, 15:14:31 UTC] {connection.py:1196} INFO - Connecting to GLOBAL Snowflake domain
[2025-02-05, 15:14:31 UTC] {connection.py:1277} INFO - This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity.
[2025-02-05, 15:14:32 UTC] {cursor.py:1156} INFO - Number of results in first chunk: 14
[2025-02-05, 15:14:32 UTC] {connection.py:788} INFO - closed
[2025-02-05, 15:14:32 UTC] {connection.py:794} INFO - No async queries seem to be running, deleting session