Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persist normalized in the same schema as unnormalized tables #845

Closed
sherifnada opened this issue Nov 7, 2020 · 3 comments
Closed

Persist normalized in the same schema as unnormalized tables #845

sherifnada opened this issue Nov 7, 2020 · 3 comments
Assignees
Labels
type/enhancement New feature or request
Milestone

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Nov 7, 2020

Tell us about the problem you're trying to solve

It is unintuitive that when a user requests data to be replicated into schema (or dataset in BQ) X that we replicate it into schema X_NORMALIZED. We should replicate the normalized table into X.

@ChristopheDuong
Copy link
Contributor

DBT is not able to overwrite the destination table that contains the JSON blob data with a new table with the proper schema since it is using that JSON BLOB as input. (Same table is simultaneously used as input and output)

So the destination connectors should write into a different table name instead (with a _raw suffix?) so that DBT can actually output the final destination table name with the proper normalized schema.

Currently to bypass this problem, DBT is writing in a different schema that has a _NORMALIZED suffix

@michel-tricot
Copy link
Contributor

I like the idea of writing to a table with the _raw suffix

@sherifnada
Copy link
Contributor Author

complete with #874

@cgardens cgardens added this to the v0.5.0 milestone Nov 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants