Handle more than 100 fields to compute hashid #970
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
michel-tricot
approved these changes
Nov 13, 2020
| hash_node_columns = jinja_call(f"dbt_utils.surrogate_key([\n {hash_node_columns}\n ])") | ||
| # Disable dbt_utils.surrogate_key for own version to fix a bug with Postgres (#913). | ||
| # hash_node_columns = jinja_call(f"dbt_utils.surrogate_key([\n {hash_node_columns}\n ])") | ||
| # We should re-enable it when our PR to dbt_utils is merged |
There was a problem hiding this comment.
Can you add the link to the dbt PR?
There was a problem hiding this comment.
The dbt-utils PR is here: dbt-labs/dbt-utils#296
michel-tricot
approved these changes
Nov 13, 2020
sherifnada
approved these changes
Nov 13, 2020
|
don't forget to publish the image once it's good to go |
jrhizor
approved these changes
Nov 13, 2020
roman-yermilov-gl
added a commit
that referenced
this pull request
Jan 19, 2023
* Source Github: handle 502 Bad Gateway with proper log message * Source Github: bump version * Source Github: update changelog for new 0.3.12 version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
etsybaev
pushed a commit
that referenced
this pull request
Jan 19, 2023
* Source Github: handle 502 Bad Gateway with proper log message * Source Github: bump version * Source Github: update changelog for new 0.3.12 version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
When running normalization on large tables, we combine all columns together via concat function and compute a hash from there to generate a surrogate key that could be used as primary key / foreign key in nested tables.
Unfortunately, the current code fails on Postgres because we cannot pass more than 100 arguments to a function for that SQL engine in order to close #913
How
Override
dbt_utils.concat&dbt_utils.surrogate_keymacros to handle more than 100 columns on postgres by using the binary||concat operator as an alternative toconcatfunction:(this should be a fix in the dbt-utils repository but we will override it for the moment)
https://www.postgresqltutorial.com/postgresql-concat-function/