Configure Airflow tasks using dbt model meta

[wornjs]

Description

The various dbt models have unique characteristics, and some may require the use of custom pools, queues, or other specific configurations. To support such cases, this update introduces the ability to add necessary information in the meta section of the dbt model.yaml. This metadata is then passed as kwargs to the corresponding Airflow tasks, enabling model-specific customization and enhanced task configuration.

here is sample

DbtTaskGroup - default_args for all dbt models

    dbt_task_group = DbtTaskGroup(
        project_config=,
        profile_config=ProfileConfig,
        default_args={'pool': dbt_pool}
    )

version: 2

models:
  - name: name
    description: description
    meta:
      owner: 'jaegwon.seo@toss.im'
      cosmos:
        operator_kwargs:
          pool: abcd

result

general pool

custom pool

Related Issue(s)

Closes: #881
Closes: #1325

Breaking Change?

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[wornjs]

It would be great to add documentation for the features introduced in this PR. However, looking at the current project, it seems there aren’t any markdown files for documentation apart from the CONTRIBUTING file. Where do you think would be the best place to add this documentation?

[pankajastro]

It would be great to add documentation for the features introduced in this PR. However, looking at the current project, it seems there aren’t any markdown files for documentation apart from the CONTRIBUTING file. Where do you think would be the best place to add this documentation?

We use rst. You can find docs at https://github.com/astronomer/astronomer-cosmos/tree/main/docs

[pankajastro]

Could you please rebase this PR

[codecov]

Codecov Report

Attention: Patch coverage is 90.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 96.21%. Comparing base (fdf7025) to head (7242dcf).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
cosmos/dbt/graph.py	86.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1339      +/-   ##
==========================================
- Coverage   96.24%   96.21%   -0.04%     
==========================================
  Files          67       67              
  Lines        4051     4070      +19     
==========================================
+ Hits         3899     3916      +17     
- Misses        152      154       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[linchun3]

related issue?
#881

[tatiana]

Hi, @wornjs,

The per-node configuration has been a long-standing request (example: #881 (comment)), and your PR solves this. This is an exciting feature; thanks for contributing to Cosmos!

A question and two requests:

1. What are the advantages/disadvantages of having these properties in meta as opposed to having them in config:

    meta:
      cosmos:
        pool: abcd

versus

    config:
      cosmos:
        pool: abcd

2. Please, could you add tests to cover this feature?

3. We'll also need docs.

If you can address these before 20 December, we can ship Cosmos 1.8. I'm tentatively adding this to that milestone.

[wornjs]

Hi, @wornjs,

The per-node configuration has been a long-standing request (example: #881 (comment)), and your PR solves this. This is an exciting feature; thanks for contributing to Cosmos!

A question and two requests:

1. What are the advantages/disadvantages of having these properties in meta as opposed to having them in config:

    meta:
      cosmos:
        pool: abcd

versus

    config:
      cosmos:
        pool: abcd

2. Please, could you add tests to cover this feature?
3. We'll also need docs.

If you can address these before 20 December, we can ship Cosmos 1.8. I'm tentatively adding this to that milestone.

i'm gonna add test until this week

[dwreeves]

Hi, responding here instead of in #1325 as it seems the discussion has migrated here.

First, this PR is related to #881. This issue was for supporting arbitrary kwargs, not just a single kwarg.

I was a big fan of where we ended up with the API of that proposal, and there are a few differences.

The first difference, as @tatiana brings up, is that it uses config: instead of meta:. I do not believe meta: is a valid attribute of a model specification. https://github.com/dbt-labs/dbt-core/blob/fc6167a2ee5291cf6c562eadcd975b48e6a34d65/core/dbt/artifacts/resources/v1/model.py#L38-L48 Please let me know if I am wrong, but I do believe it should be config: and not meta: based on the dbt-core source code. I am wrong, it is! There is a chain of inheritances that lead to meta. My mistake. With that out of the way, I am not sure what I prefer between config and meta, to be honest. I would ask the same question as Tatiana again then, i.e. what do you see as the difference between one or the other and why should one (in this case meta) be preferred?

The second difference is, as mentioned, it supports arbitrary kwargs, not just pool. There are a lot of reasons why someone may want to inject many different kwargs into their operators, not just pool.

The third difference is that the namespace was cosmos: operator_args: pool:, not just cosmos: pool:. I think this is really important once you move the API from just pool to any kwarg. The issue is, in the future, we may want to add additional options that are not actually kwargs (or alternatively allow users to inject such things themselves), so we would need to keep the kwargs separate of things that are not kwargs.

So my final proposal would be this:

version: 2
models:
  - name: model_a
    config:
      alias: model_a
      cosmos:
        operator_args:
          pool: my-pool-here

(or alternatively, if using meta:)

version: 2
models:
  - name: model_a
    config:
      alias: model_a
    meta:
      cosmos:
        operator_args:
          pool: my-pool-here

^ So the key difference is there is one extra dict, i.e. operator_args.

But really, I think we should just **unpack the whole cosmos.operator_args into the operator so that users can do things like this:

version: 2
models:
  - name: model_a
    config:
      alias: model_a
      cosmos:
        operator_args:
          pool: my-pool-here
          retries: 4
          conn_id: special_conn_id

[dwreeves]

I've confirmed both meta and config have the same merge behavior: both merge the top-level keys, but neither recursively merges, which is annoying because

1. it makes it harder to decide whether cosmos should be in meta: or config:
2. it also means there is no way for users to do something like +config: cosmos: operator_args: x: ... in dbt_project.yml and config(cosmos={"operator_args": "y": ...}) in a sql model and get both x and y 🫤

(The only way to support merging this way would be to use top-level attributes, but I think that's tedious and limiting.)

There is an issue open relating to this in dbt-core where someone suggests recursive merging for meta: dbt-labs/dbt-core#10946

[wornjs]

I thought of config as settings related to dbt (e.g., materialized, tag), and meta as a way to display additional information beyond that, so I used the values in meta. Although config and meta are somewhat similar, I will follow the agreed-upon conventions.

[wornjs]

@tatiana
sorry for late commit plz check this

[tatiana]

@wornjs, thank you very much fo adding tests and documentation. Your work will be one of the main features of Cosmos 1.8.

Based on what you and @dwreeves confirmed, there is little difference between config and meta in dbt; I'm happy with using meta, as you implemented. Thank you for checking/explaining.

Please address only one last thing I'd like you to address before we release this feature (if possible, by 20th December). The request is to have, within the "cosmos" meta config, a "operator_args" key. It is a minor change of interface from what you originally proposed:

version: 2

Models:
  - name: name
    description: description
    meta:
      owner: 'jaegwon.seo@toss.im'
      cosmos:
        operator_kwargs:
          pool: abcd

This suggestion, originally from @dwreeves, can be beneficial as we expand the usage of this feature. Let's say, for instance, we'd like users to be able to configure the ExecutionMode by dbt model. This is not an operator kwarg, so having the operator_args key would allow us to differentiate between properties that may affect a model execution but are not necessarily arguments passed to the operator itself.

[wornjs]

@tatiana
i fix it plz check ^^

[tatiana]

Thank you very much @wornjs for improving Cosmos and adding this feature.

I can't wait to hear the feedback from the community. Given the release timeline, I'll merge even though the coverage could be improved - we can improve this in the future.