Make dbt-sql and default-sql templates public

[lennartkats-db]

Changes

This makes the dbt-sql and default-sql templates public.

These templates were previously not listed and marked "experimental" since structured streaming tables were still in gated preview and would result in weird error messages when a workspace wasn't enabled for the preview.

This PR also incorporates some of the feedback and learnings for these templates so far.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 7 lines in your changes missing coverage. Please review.

Project coverage is 53.57%. Comparing base (e22dd8a) to head (2deaa71).
Report is 128 commits behind head on main.

Files	Patch %	Lines
libs/template/materialize.go	0.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1463      +/-   ##
==========================================
+ Coverage   52.25%   53.57%   +1.31%     
==========================================
  Files         317      351      +34     
  Lines       18004    20271    +2267     
==========================================
+ Hits         9408    10860    +1452     
- Misses       7903     8612     +709     
- Partials      693      799     +106     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pietern]

This will also apply to the prod target, which contradicts the explanation in the template:

yes, use a schema based on the current user name during development

You could use target overrides to only apply this for the dev target.

[lennartkats-db]

This should actually work as intended since the "dev_schema" var is only used in the dev profile: https://github.com/databricks/cli/pull/1463/files#diff-0ca9390ffa1ff34b28194f7947d477dd79b02f9fe6aa14a578483469913e4d9aR21.

This is the place where our dev/prod targets and the dbt profiles come together. Which can look a bit weird. To make the logic of the generated yml seem bit simpler, I'm no longer passing "dev_schema" unless yes, use a schema based on the current user name during development is actually enabled. Once that is enabled, customers will have to stare at the slightly more complicated ymkl.

[pietern]

It is possible to do this exclusively for the dev target using overrides. Then it is more clear the dev schema only pertains to the dev target. It's pretty confusing as is, IMO.

targets:
  dev:
    resources:
      jobs:
        dbt_project_job:
          tasks:
            - task_key: dbt
              dbt_task:
                commands:
                  - 'dbt deps --target=${bundle.target}'
                  - 'dbt seed --target=${bundle.target} --vars "{ dev_schema: ${workspace.current_user.short_name} }"'
                  - 'dbt run --target=${bundle.target} --vars "{ dev_schema: ${workspace.current_user.short_name} }"'

  prod:
    resources:
      jobs:
        dbt_project_job:
          tasks:
            - task_key: dbt
              dbt_task:
                commands:
                  - 'dbt deps --target=${bundle.target}'
                  - 'dbt seed --target=${bundle.target}'
                  - 'dbt run --target=${bundle.target}'

The top level resource then shouldn't specify commands, as lists are appended in overrides.

[pietern]

Or define another variable that interpolates the inner string for --vars.

[lennartkats-db]

The challenge with overrides is that it duplicates things and breaks visual editing. And in this case, we're just passing the value of a parameter, not changing behavior. So I'd rather just keep a single version of this job.

[pietern]

How about a comment then? The generated configuration should pre-empt questions people have. Maybe "the dev schema is not used in the prod target" or something?

[lennartkats-db]

Yes that sounds good, added!

[pietern]

Right now the workspace host is displayed as if it were a variable. The intent is for this to be human-readable right?

[lennartkats-db]

@pietern

Right now the workspace host is displayed as if it were a variable. The intent is for this to be human-readable right?

I'm not sure what you mean? It doesn't look like a variable on your screenshot? Do you mean that it looks a bit like an input prompt? That is intentional to make all settings appear similar.

[pietern]

I would expect it to read like prose, like the preceding text.

The template targets workspace https://...

As opposed to a key: value.

[lennartkats-db]

@pietern yeah, it totally could. My mental model has been that this is just another setting but one that is set indirectly. That seemed like the easiest way to explain it. Any suggestions for phrasing if we present that in a different way?

I tried

Welcome to the dbt template for Databricks Asset Bundles!

Workspace selected based on your current profile: https://e2-dogfood.staging.cloud.databricks.com
(See https://docs.databricks.com/dev-tools/cli/profiles.html for how to change this).

Please provide a unique name for this project.
project_name [dbt_project]:

But the text just seems a bit out of place

[pietern]

Is the target not required after running deps?

[lennartkats-db]

Ah yes, it is. Good catch. As you can see it is used in the other branch of the {{if}}.