[Feature] dbt compile to generate as well "run" queries

[Charles1104]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

dbt compile currently only generates compiled queries within the "target/compiled" directory. Those queries are the compiled "as is" of the queries written in your models. However those queries do not include "pre-hooks", "post-hooks", nor the DML wrapper that dbt includes in order to create the table or incrementally add new records. For instance for a BigQuery model, it might add the following on top of the original query:

create or replace table [database].[schema].[table] partition by [date field] cluster by [fields] OPTIONS( description="""to_fill""" ) as (

Those "run" models are as valuable as the compiled ones in case you are orchestrating your queries without using dbt run commands. You can then for instance have a BigQuery operator on Airlfow that then points to the "DBT run" queries.

End goal:

dbt compile should generate "compiled" and "run" queries.

Describe alternatives you've considered

/

Who will this benefit?

This will benefit anyone that is running dbt on an orchestrator and that wants to run the queries without the dbt run command but via the orchestrator operator instead.

Are you interested in contributing this feature?

/

Anything else?

No response

[jtcohen6]

Hey @Charles1104, thanks for opening!

There was another issue about this a few months ago, recently closed for being stale: #3222. If you don't mind me paraphrasing myself for a bit:

There's a subtle distinction today between a model's:

• Compiled SQL: What appears in the docs (via manifest.json), and in target/compiled. This is the result of simply compiling the Jinja + SQL from within a model file. It includes conditional logic templated into model SQL, e.g. is_incremental(), but it does not include any other materialization logic.
• Run SQL: What appears in the logs, and in target/run. It includes the full materialization logic, in this case, the output of macros which the incremental materialization calls, and to which it passes the model's compiled SQL as an argument.

Today, dbt isn't able to know the "run SQL" for a model until it is running it. The materialization code is interpreted dynamically. Some information really is hard to know in advance, if it depends on database state or dynamic introspective queries, but most is; I fully agree that dbt should be able to know it, even make it possible to "dry run" a model (#1059). This will require a significant rework of materializations, something I've wanted for a very long time. It's possibile we tackle that after v1—so, next year at the very earliest.

[Charles1104]

Hi @jtcohen6

Thank you for the answer and for the detailed explanation.

The "dry-run" flag on a model would be awesome and I will follow on #1059 and try to involve my team to collaborate on it.

Best regards,

Charles

[jtcohen6]

Closing this now, to keep the conversation centralized in the related issue