Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-3299] [SL] Support Hierarchical Configs for Saved Queries and their Exports #8956

Closed
1 task done
QMalcolm opened this issue Oct 31, 2023 · 0 comments · Fixed by #9065
Closed
1 task done

[CT-3299] [SL] Support Hierarchical Configs for Saved Queries and their Exports #8956

QMalcolm opened this issue Oct 31, 2023 · 0 comments · Fixed by #9065
Assignees
Labels
backport 1.7.latest Impact: SL semantic Issues related to the semantic layer user docs [docs.getdbt.com] Needs better documentation

Comments

@QMalcolm
Copy link
Contributor

QMalcolm commented Oct 31, 2023

Housekeeping

  • I am a maintainer of dbt-core

Short description

In #8950 we added Exports to SavedQueries. Exports have configs and we want these configs to be setable from a project's dbt_project.yml, i.e. hierarchical. Originally this work was to be part of #8892 in #8950, however due to deadlines and some pesky edge cases, we separated out this work. With that said, we've thought through (and tried some) approaches

  1. Write an entire bespoke config compiler just for Exports
  2. Make an Export a node type (but don't put it in the DAG), and then use calculate_node_config to generate the config
  3. Abstract pretty much all of calculate_node_config except the config class getter into a separate function which would then be called by calculate_node_config and directly from the export parsing

We haven't attempted option (1) because it seemed wrong to write a whole bunch of bespoke code for this. Options (2) and (3) suffer from an issue where-in the ExportConfig has a schema_name attr, but the UnparsedExportConfig uses schema (this is necessary for other reasons). Additionally option (2) would make an export a pseudo node, which we don't want to do unintentionally.

As it stands it seems like there are two paths forward: option (3) but use the UnrenderedExportConfig during the config generation process and then after the fact map things to the ExportConfig or a forth option would be to refactor our config parsing process to be less clever.

Acceptance criteria

We have implemented a way for Export configs to be set hierarchically AND it is tested

Impact to Other Teams

Semantic Layer, maybe also Cloud Artifacts?

Will backports be required?

Yes 1.7

Context

No response

@QMalcolm QMalcolm added the user docs [docs.getdbt.com] Needs better documentation label Oct 31, 2023
@github-actions github-actions bot changed the title [SL] Support Hierachegal Configs for Saved Queries and their Exports [CT-3299] [SL] Support Hierachegal Configs for Saved Queries and their Exports Oct 31, 2023
@QMalcolm QMalcolm changed the title [CT-3299] [SL] Support Hierachegal Configs for Saved Queries and their Exports [CT-3299] [SL] Support Hierarchical Configs for Saved Queries and their Exports Oct 31, 2023
@QMalcolm QMalcolm self-assigned this Nov 1, 2023
@QMalcolm QMalcolm added semantic Issues related to the semantic layer Impact: SL labels Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.7.latest Impact: SL semantic Issues related to the semantic layer user docs [docs.getdbt.com] Needs better documentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant