Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat: Snowflake: Handle forward_only changes to 'clustered_by' #3205

Merged
merged 2 commits into from
Oct 1, 2024
Merged

Feat: Snowflake: Handle forward_only changes to 'clustered_by' #3205

merged 2 commits into from
Oct 1, 2024

Conversation

erindru
Copy link
Collaborator

@erindru erindru commented Oct 1, 2024

This change allows you to update clustered_by on a Snowflake model, trigger a forward-only plan and have the cluster key on the table be altered.

Previously, it would be a no-op which was a bit misleading

izeigerman
izeigerman reviewed Oct 1, 2024
View reviewed changes
sqlmesh/core/engine_adapter/snowflake.py
@@ -422,3 +433,49 @@ def _create_column_comments(
f"Column comments for table '{table.alias_or_name}' not registered - this may be due to limited permissions.",
exc_info=True,
)

def get_alter_expressions(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something we can only do for snowflake?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now, I didn't investigate the possibility or syntax on other databases

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we please follow-up with the same support for BigQuery

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3231

izeigerman
izeigerman reviewed Oct 1, 2024
View reviewed changes
sqlmesh/core/engine_adapter/snowflake.py Outdated
actions=[exp.Cluster(expressions=parsed_cluster_key.expressions)],
)
)
else:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we can do elif current_table_info.is_clustered: here and get rid of the any block

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair call, i've updated it

@erindru erindru merged commit e099cbb into main Oct 1, 2024
23 checks passed
@erindru erindru deleted the erin/snowflake-clustered-by branch October 1, 2024 20:11
@erindru erindru mentioned this pull request Oct 8, 2024
Merged
justinjoseph89 added a commit to trygforsikring/sqlmesh that referenced this pull request Oct 21, 2024
@justinjoseph89 @benfdking
@georgesittas @themisvaltinos @erindru @treysp @z3z1ma @tobymao @izeigerman @vchan @VaggelisD @Harmuth94 @Kayrnt @crericha @eakmanrq
Remove sink and make it insync with base (#6)
f02a437 
* Fix: mark kind changes as breaking in forward only plan (TobikoData#3207)

* Feat: add support for parameterized python model names (TobikoData#3208)

* Fix: Bigquery support of complex nested types (TobikoData#3190)

* Feat: Snowflake: Handle forward_only changes to 'clustered_by' (TobikoData#3205)

* Docs: add gateway variables to jinja macros concepts doc (TobikoData#3210)

* Fix: avoid parsing column names into qualified columns in InsertOverwriteWithMergeMixin (TobikoData#3211)

* Chore: bump sqlglot to v25.24.2 (TobikoData#3213)

* Feat: Support INCREMENTAL_BY_TIME_RANGE models on Athena/Hive (TobikoData#3201)

* Fix: load custom materializations on run (TobikoData#3216)

* Fix: Infer column types when data type is omitted in dbt seeds (TobikoData#3215)

* Chore: bump sqlglot to v25.24.3 (TobikoData#3217)

* Fix: DBT seed column order (TobikoData#3221)

* fix: web reloading caused iteration error (TobikoData#3220)

* Fix: Make dbt adapter macros available in the local scope (TobikoData#3219)

* Feat: Support DBT Athena adapter (TobikoData#3222)

* chore: docs

* Feat: Support SQLMesh project generation from dlt pipeline (TobikoData#3218)

* Fix: Broken hive distro link in the test airflow image

* Fix: Prevent loaded context from being used concurrently (TobikoData#3229)

* Fix: Go back to using hive 3.1.3 for the Airflow test image

* Fix: Support of custom roles for Postgres (TobikoData#3230)

* Fix(redshift): regression in varchar length workaround (TobikoData#3225)

* Fix: Force the CircleCI's git to use https links when running pre-commit (TobikoData#3235)

* Fix: reset macro registry *after* loading models (TobikoData#3232)

* Fix: Modify dlt query filter not to use alias reference (TobikoData#3233)

* Fix: Support CLUSTER BY clause for the Databricks engine (TobikoData#3234)

* Feat: BigQuery - Handle forward_only changes to clustered_by (TobikoData#3231)

* chore: Fix typo in model_kinds.md (TobikoData#3239)

* Feat: support custom unit testing schema names (TobikoData#3238)

* Chore: Make the scheduler config extendable (TobikoData#3242)

* Fix: use parentheses for databricks' CLUSTER BY clause (TobikoData#3240)

* Fix: handle Paren in depends_on validator (TobikoData#3243)

* fix: data diff for bigquery project parsing (TobikoData#3248)

* Chore: Reintroduce parallelism in integration tests (TobikoData#3236)

* Feat(databricks): Add OAuth support (TobikoData#3250)

* Chore!: bump sqlglot to v25.25.0 (TobikoData#3252)

* Adding markdown feature to model description (TobikoData#3228)

* Fix: refactor table part parsing for Snowflake (TobikoData#3254)

* Fix: always warn when an audit has failed (TobikoData#3255)

* Chore: bump sqlglot to v25.25.1 (TobikoData#3256)

* Ensure using project instead of execution project for temp table as default (TobikoData#3249)

* Chore: Clarify that restatement plans ignore local changes (TobikoData#3257)

* feat!: run-all bot command errors if anything within it errors (TobikoData#3262)

* Fix(clickhouse): remove fractional seconds when time column is datetime/timestamp type (TobikoData#3261)

* remove risingwave configuration from dbt

* remove sink settings

* remove risngwave sink

* introducing risingwave as state syn engine

* add risingwave connetion as test

* change test case

* Fix: Prevent extraction of dependencies from a rendered query for dbt models (TobikoData#3263)

---------

Co-authored-by: Ben <9087625+benfdking@users.noreply.github.com>
Co-authored-by: Jo <46752250+georgesittas@users.noreply.github.com>
Co-authored-by: Themis Valtinos <73662635+Themiscodes@users.noreply.github.com>
Co-authored-by: Erin Drummond <erin.dru@gmail.com>
Co-authored-by: Trey Spiller <1831878+treysp@users.noreply.github.com>
Co-authored-by: Alexander Butler <41213451+z3z1ma@users.noreply.github.com>
Co-authored-by: Toby Mao <toby.mao@gmail.com>
Co-authored-by: Iaroslav Zeigerman <zeigerman.ia@gmail.com>
Co-authored-by: Vincent Chan <vchan@users.noreply.github.com>
Co-authored-by: Vaggelis Danias <daniasevangelos@gmail.com>
Co-authored-by: Harmuth94 <86912694+Harmuth94@users.noreply.github.com>
Co-authored-by: Christophe Oudar <kayrnt@gmail.com>
Co-authored-by: Chris Rericha <67359577+crericha@users.noreply.github.com>
Co-authored-by: Ryan Eakman <6326532+eakmanrq@users.noreply.github.com>
Co-authored-by: Justin Joseph <justin.joseph@tryg.dk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@izeigerman izeigerman izeigerman approved these changes

@georgesittas georgesittas georgesittas approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@erindru @izeigerman @georgesittas