Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First version of distributed materialization #15

Closed
wants to merge 7 commits into from

Conversation

Volodin-DD
Copy link

NOTE: lightly tested (though successfully). Need help from community.

Made two materializations: distributed_table for distributed version of table models and distributed_incremental for incremental models.

The main idea is to make local tables on cluster with engine and parameters from config in model's sql file within a distributed table "looking" on these local tables. So when referencing model with distributed materialization dbt will make insert into distributed table and select from distributed table. Sharding key is also to be set on model's config.

Main flow described in comments distributed_incremental.sql

@Volodin-DD
Copy link
Author

There were some issues with schemas. Fixed it

@CLAassistant
Copy link

CLAassistant commented Jun 5, 2022

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ Volodin-DD
❌ Dimitry Volodin


Dimitry Volodin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@genzgd
Copy link
Contributor

genzgd commented Nov 30, 2023

Closed in favor of #172 and #186

@genzgd genzgd closed this Nov 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants