Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Support merge update operation #8685

Closed
ZhiHanZ opened this issue Nov 7, 2022 · 4 comments
Closed

Feature: Support merge update operation #8685

ZhiHanZ opened this issue Nov 7, 2022 · 4 comments
Labels
C-feature Category: feature

Comments

@ZhiHanZ
Copy link
Collaborator

ZhiHanZ commented Nov 7, 2022

Summary
compare and update target table based on reference table

example:

merge into target_table using source_table 
    on target_table.id = source_table.id
    when matched then 
        update set target_table.description = source_table.description;
+------------------------+
| number of rows updated |
|------------------------|
|                      1 |
+------------------------+

Ref:
https://docs.snowflake.com/en/sql-reference/sql/merge.html

@ZhiHanZ ZhiHanZ added the C-feature Category: feature label Nov 7, 2022
@hantmac
Copy link
Member

hantmac commented Nov 29, 2022

dbt snapshot depends on merge into.

@ZhiHanZ
Copy link
Collaborator Author

ZhiHanZ commented Nov 30, 2022

dbt snapshot depends on merge into.

dbt adapter documentation suggests that we could use delete + insert to replace merge operations. I think that may worth trying

@flaneur2020 flaneur2020 self-assigned this Feb 3, 2023
@flaneur2020
Copy link
Member

flaneur2020 commented Feb 3, 2023

I'd like to take this issue to have an investigation about whether it's hard or not to implementing the MERGE INTO statement in our code base.

An investigation is prioritized first, the implementation may be seperated in serveral issues.

Nowadays we have already got the UPDATE support, I guess MERGE may have similiar behaviours: they both find out which blocks to update, then make modifications among these blocks, and write the blocks back to storage.

@flaneur2020 flaneur2020 removed their assignment Feb 6, 2023
@flaneur2020
Copy link
Member

flaneur2020 commented Feb 6, 2023

MERGE statement have some complexities about handling JOINs. To make the CDC solutions, making UPSERT may be a easier path: #9861

the UPSERT statement do not need integration with JOIN, and can process batched CDC streams.

but UPSERT have some limitations when compared with the MERGE statment:

  • can not run some basic ETL (like extract fields from json) in the SQL like the merge statement
  • can not handle DELETE

however, we can do MERGE in the future.

@zhyass zhyass removed their assignment Oct 19, 2023
@ZhiHanZ ZhiHanZ closed this as completed Nov 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature
Projects
None yet
Development

No branches or pull requests

4 participants