Feature: dbt integration

[oscar60310]

Using the result of dbt (dbt build) as API schemas.

https://www.notion.so/canner/Integration-with-dbt-61427c0571e04d82b2696f4cc50f3ff1

Feature description

DBT

1. Create some models and write some dbt models.

   --- models/example/1_table.sql
   
   {{ config(materialized='table') }}
   
   with source_data as (
       select 1 as id, 'Ivan' as name, 18 as age
   	UNION
       select 2 as id, 'William' as name, 80 as age
       UNION
       select 3 as id, 'Eason' as name, 18 as age
   )
   
   select * from source_data

2. Run dbt models: dbt run

Vulcan

1. Install DBT extension. yarn add @vulcan-sql/extension-dbt.

2. Config extension set the path to dbt manifest files.

   extensions: ['@vulcan-sql/extension-dbt']
   templateEngine:
     dbt:
         modelFiles:
         - /path/to/manifest.json

3. Reference dbt model in your SQL. (model.[project-name].[model-name])

   select * from {% dbt "model.test.1_table" %}

Result

Results are variavous by materialization

-- view table incremantal
select * from "postgres"."public"."1_table"
-- ephemeral (sub-query)
select sub.* from (
select *
from "postgres"."public"."1_table"
where age <= 18) as sub

Implementation

1. I created two packages: extension-dbt for the extension and test-utility as a collection of testing tools like creating mock compilers ...etc.
2. Add a hook "onInit" for template engine extension to execute some asynchronous jobs.
3. Turn compiler.compile() function to async to wait onInit hooks.

[kokokuo]

excellent, really easy to understand, LGTM 👍

[kokokuo]

NIT: Maybe we could add the comment to mention other members to notice here, if not found relation_name or relation_name is null, then will read compiled_sql value, how do you think?

[oscar60310]

Sure, I'll account for the logic here.

[oscar60310]

Hi @kokokuo , all issues have been fixed.