Allow using macros in the documentation YAML

[pcasteran]

Describe the feature

Currently, only the doc macro is available in the Jinja rendering context used for the doc generation.
It would be useful to add all the macros available in the project, as it is the case during the compilation of the models.
The use-case I'm trying to implement is to inherit the columns' description between all the model layers without having to copy-paste them (see this comment).

Describe alternatives you've considered

See the discussion on 2995.

Additional context

I have looked at the code and the reason why we can't use macros in the documentation comes from the fact that the DocsRuntimeContext class inherits from SchemaYamlContext whereas the context used during model compiltation inherits from ManifestContext which is responsible for adding the macros in the context used during Jinja rendering.

As these classes are not documented it's quite difficult to figure out their responsibilities and the reasons behind this type hierarchy. But maybe modifying the SchemaYamlContext to inherit from ManifestContext would do the trick and enable the use of macros during the processing of the documentation.

Who will this benefit?

Anyone who wants to write "advanced" documentation using some cool features of dbt would benefit from this feature.
Inheriting column's description and test from previous models is one example. There would also be other usages for advanced formatting of the documentation.

Are you interested in contributing this feature?

I would be interested in contributing but I would need some pointers on how to achieve this.

[jtcohen6]

@pcasteran I really appreciate you opening this issue. I am going to ask you to be more specific :)

Anyone who wants to write "advanced" documentation using some cool features of dbt would benefit from this feature.
Inheriting column's description and test from previous models is one example. There would also be other usages for advanced formatting of the documentation.

What other advanced use cases do you have in mind? If we're both thinking that the primary use case would be extending/inheriting descriptions, as is described by #2995, then I'll share how I'm thinking about that problem, and why this isn't my preferred solution (at least for now).

Problem: There's a lot of duplicated code in projects today, defining resource properties on one object that apply equally to another object.

Desired future state: Users can extend resource properties from one model/column/etc to another, without:

• resorting to copy-paste
• needing to define those objects within the same file (a limitation of vanilla YAML anchors)

With that in mind, let's think about this specific proposal: expanding the context available when dbt renders description fields, to include many more user-defined macros.

Pros:

• It would be fairly straightforward to implement: dbt already uses Jinja rendering for description fields. The work would all be around the Jinja context available for that rendering.

Cons:

• By and large, dbt only Jinja-renders one property at a time. As such, this approach would not do a lot to help with the full use case of extending/inheriting multiple properties from one model to another: columns, descriptions, tests, tags, meta properties, etc. For this capability, we should be looking at solutions in YAML, not Jinja.
• Performance. Because Jinja is involved, description rendering is one of the slowest steps in dbt "mise-en-place" today, which is something we're working hard to speed up. Adding arbitrarily more macros to the rendering context risks slowing this down significantly.

So: I want to hear what other use cases you're thinking about. I agree completely with the desire for inheritable/extensible descriptions—I just don't think this is the right way there.

[djbelknapaw]

This adds a level of complexity, but it would be ideal if this ignored macros or functions inside a code tag within the markdown as well. For example, dbt tries to run this ref() right now when it's inside a code tag:

-- depends_on: {{ ref('my_model') }}

but it would be preferable if that code block was left as-is. Maybe this is just a nice-to-have after a first pass enabling more functionality.

I didn't open this issue but I'm interested in it as well. I really like the suggestions going on in #2995, and #1334 describes my original use case even though I didn't open that one either! Overall, having access to more functions/macros would be handy.

I'd love to contribute but I'm not at a point where I'd be very useful just yet on back-end stuff. And maybe the code-tag thing is a separate issue, I can split it out if that's helpful.

[pcasteran]

Thanks for taking the time to look at my proposal and for your detailed answer @jtcohen6.

First, I want to clarify the scope of this discussion. What I'm actually proposing is to enable more macros to be used during the doc rendering, it is not a proposal for documentation inheritance. I may plan to use it for this use-case in the short term but the proposal was more general than that.
If we go back to the initial topic, is there a reason why macros aren't currently supported (except doc) during the doc generation or is it just because there were simply no use for it yet?

Besides column doc inheritance, I would also like to access some attributes of other models in my documentation like {{ ref('...').path.identifier }} to display the actual table name in which is materialized the model.

--

Now regarding the more specific topic of documentation reuse, there are many possible ways to achieve it and maybe there should not be only one implemented.
Indeed the solution I described in #2995 can't be the "official" recommended way to do it, but only one made possible by allowing to use macros during the doc processing. I totally agree it cannot be a complete solution as it does not allow inheriting blocks of columns or anything else than column description.

Performance wise, I'm not sure there would be a penalty using this kind of macro as, if I remember correctly the code I saw, dbt caches the columns description and only performs Jinja rendering once per column.

--

Regarding the use of YAML anchors, is there a discusion about it other than #1790 ? I'm not really convinced this is the way to go as it raises some concerns:

1. Basically it means creating a parallel model dependency processing logic beside the parsing already done by dbt.
2. It induces a dependency on the location/structure of the documentation inside the project instead of relying on the model graph resulting from the parsing.
3. Will it work when refactoring (fusing or merging) some YAML files without having to modify the ones depending on it?
4. Will it work for models from another projects imported as packages?

[jtcohen6]

@pcasteran Apologies for the delay in getting back to you! This is something I continue to think about, and I'd like to find the right way to do it, possibly next year. For now, this has to wait until after dbt-core v1.0, which is our top development priority for dbt-core for the rest of the year.

A similar idea just came in from @brittianwarner over in #3827, with the specific thought of more dynamic support in docs blocks (including "special" ones like __overview__). This got me thinking about the potential of expanded rendering contexts for descriptions, in lieu of arbitrary macro support. Those contexts could include things like {{ model }}, {{ column_name }}, vars, env vars, etc. Those make a lot of sense in a world with inherited documentation, and they'd get you the other piece you mention ({{ ref('...').path.identifier }}), while still preserving the guardrails we have about parse-time documentation rendering today.

[octocat1000]

Just came across a similar problem when trying to add dynamic content to the docs. I'd just assumed I could use macros in my docs. Found out otherwise then found this issue. My use case is that I'm trying to insert some data quality metrics into my docs by running a query and using the results in the docs. I'd like to be able to keep my query as a macro (organized with my macros for models) and then have it be called when I'm running docs generate to fill in the results.

[jmg-duarte]

Hey all, I was looking into using macros to simplify some of my documentation blocks. Want I want is to be able to define a macro like:

{%- macro client_example(table_name) -%}
```py
df = client.{{table_name}}.collect()
print(df.sum())
```
{%- endmacro -%}

And be able to use it in the description field of tables like:

table:
  - description: | 
      {{doc(...)}}
      Code example:
      {{client_example("current_table")}}

From an end-user perspective, it doesn't seem to be a complicated feature to add, but I'm sure there is something behind the scenes that is stopping this from progressing.

I've found several discussions around the topic (#3827, #2995, and this one) but none has had a clear outcome.

Is there a place summarizing the status of this feature?

[jose-workpath]

In my case, I would like to see this to avoid repeating logic in macros and the YAML file with our sources. In our data warehouse, we have different source schemas for production and development, right now we are doing something like this:

version: 2
sources:
  - name: "{{ target.schema }}"
    schema: |
        {%- if env_var("DBT_ENVIRONMENT", "dev") == "dev" -%}
              dev_schema
        {%- else -%}
             prod_schema
    ...

The problem is we also have to do the same logic in some hooks, and I would really like to just have one macro that applies this logic wherever is needed (keeping it DRY).

Ideally, it would be as simple as:

version: 2
sources:
  - name: "{{ target.schema }}"
    schema: "{{ get_source_schema_from_env() }}"
    ...

[wtfzambo]

I'm facing the EXACT same situation.

[timvw]

These days we simply have many environment variables... and add the logic/conventions in the .envrc/script to calculate the values..

In this case we'd have a DBT_TARGET_SCHEMA environment variable and add the if/else logic in shell script instead..

[wtfzambo]

The way I solved this is by writing my sources.yml file like this:

version: 2
sources:
  - name: raw_my_source
    schema: |-
      raw_my_source
      {%- if target.name == "prod" -%} _production
      {%- else -%} _staging
      {%- endif -%}

And then having a CLI command that generates the file and adds that pattern automatically.

[mike-weinberg]

@jtcohen6 out of curiosity can this issue be consolidated with others? I see it's getting stale but I suspect the intent of this proposal deserves an LOE assessment

i feel like the approach doesn't have to be arbitrary macros, since much of the value in the env switching use case could be solved by baking a concept of environments/targets more deeply into the source function via an optional env parameter, a strong opinion about namespaces, and the ability to override that opinion with a specific macro, similar to how the custom database name macro works. the macro would therefore modify how source renders.

But it would be nice to be able to call macros (or callable python functions to avoid parsing) in the yml file, so long as there are clear rules on what those macros can do, and ideally, dedicate a new top level directory called something like /init containing macros which are loaded at startup before metadata or sql are parsed, so that all visitors to the project can immediately check that folder to see how the project's behavior differs from a vanilla configuration.

[mike-weinberg]

@drewbanin @jtcohen6 is it at all possible to get eyes on this? the solutions proposed here are functional but not at all preferable.

I'd really like to have some sort of special directory in which I can define macros which can be used in yaml files. eg generate_custom_database_name seems like it should be usable here for unified dispatch logic, OR perhaps it could be baked into the source function? either way, compositional dynamism would be super nice.

[karenderer]

Hi, just want to echo the above. I have a similar use-case as the above example where I want to be able to set the database/schema based on target values. I think for now I will pass a variable to get the functionality I need, but would much prefer to be able to do this via a macro.

[mroy-seedbox]

Rather than passing a variable (from the CLI/DBT command I assume), you could just register your variables directly in your dbt_project.yml.

Something like this:

vars:
  current_schema: "schema_{{ 'production' if target.name == 'prod' else 'staging' }}"

[mroy-seedbox]

Another alternative could be to override the generate_schema_name or generate_database_name macros (see documentation here).

[karenderer]

I want to dynamically change a source database so I don't think the overrides are a good fit for my use case. Is there a way to fit it neatly in one line if I have some elif conditions?

[mroy-seedbox]

Yup! You might also be able to do it directly in your general DB configs in dbt_project.yml:

models:
  my_project:
    source_db:
      +database: "{{ 'dev_source_db' if target.name[0:3] == 'dev' else 'prod_db' }}"

[mroy-seedbox]

For the elif, they are unfortunately not supported in one-liners. But you could chain multiple ifs (although it can get a bit messy):

var = 0 # try with 0, 1, or 2
print((123 if var == 1 else 456) if var > 0 else 789)

[ismailsimsek]

Hi all, want to leave link to opendbt project here. Where you can easily add this kind of features to dbt-core in an elegant way. Feel free to check it out and add the features you like to see in dbt-core