can't seem to inject vars via operator_args

[set5think]

Hi, I'm running cosmos 1.1.1, and am reading the docs and code, and am using basically the same code minus values, but no matter what I do, the dbt command is never generates with the dbt vars I pass through operator_args. When I inspect cosmos/graph.py, I don't see where operator_args is ever used to inject settings from it into the dbt the command. I only see it to populate a class called LegacyDbtProject and then just in the class signature.

Is there a different way to inject dbt vars not using operator_args?

Thanks,

-Hassan

[tatiana]

HI @set5think , thanks for posting the issue!

Have you tried the following: https://astronomer.github.io/astronomer-cosmos/configuration/operator-args.html?

Please give a step-by-step for us to reproduce the issue.

[fabiomx]

Hi,

I have the same issue. I can reproduce it with DBT_LS load mode.

operator_args = {
    "vars": {"SNOWFLAKE_DATABASE": database}
}


def get_render_config(schedule_tag: str):
    return RenderConfig(load_method=LoadMode.DBT_MANIFEST,
                        test_behavior=TestBehavior.AFTER_EACH,
                        select=[f"tag:{schedule_tag}"]
                        )


@dag(dag_id="dbt_hourly", schedule_interval=timedelta(hours=1), default_args=default_args, catchup=False)
def dbt_hourly_dag():
    start = EmptyOperator(task_id="start")

    dbt_tg = DbtTaskGroup(
        project_config=project_config,
        profile_config=profile_config,
        execution_config=execution_config,
        default_args=default_args,
        operator_args=operator_args,
        render_config=get_render_config("hourly")

    )

    end = EmptyOperator(task_id="end")

    start >> dbt_tg >> end

I get:

Broken DAG: [/home/airflow/gcs/dags/dbt_cosmos.py] Traceback (most recent call last):
  File "/opt/python3.8/lib/python3.8/site-packages/cosmos/dbt/graph.py", line 132, in load
    load_method[method]()
  File "/opt/python3.8/lib/python3.8/site-packages/cosmos/dbt/graph.py", line 255, in load_via_dbt_ls
    raise CosmosLoadDbtException(f"Unable to run dbt ls command due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run dbt ls command due to the error:
�[0m20:27:11  Running with dbt=1.5.4
�[0m20:27:19  Registered adapter: snowflake=1.5.4
�[0m20:27:19  Unable to do partial parsing because saved manifest not found. Starting full parse.
�[0m20:27:31  Encountered an error:
Compilation Error
  Could not render {{var('SNOWFLAKE_DATABASE')}}: Required var 'SNOWFLAKE_DATABASE' not found in config:
  Vars supplied to <Configuration> = {}

With DBT_MANIFEST, it works fine.

Also, if I set the var in the dbt_project file, I have no issues*, but I need to generate the var dynamically and use it via the vars argument.

*Well, I have the time-out issue mentioned here: #520 (comment), but no more var-related issues.

I'm on Cosmos 1.2.1 / dbt-core 1.5.4, on Google Cloud Composer, local execution mode.

[tatiana]

@fabiomx The issue is that the operator args is only used during task execution, not DAG parsing.

To solve the issue you described, we'll need to add vars to the RenderConfig:
https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config

Since you're not blocked (LoadMethod.MANIFEST works), we'll aim to solve this in the 1.3.0 release. Please let us know if you'd like to make this contribution!