Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for adding columns to snapshots #477

Merged
merged 5 commits into from
Mar 19, 2024
Merged

Conversation

ericmuijsvanoord
Copy link
Contributor

Fix to be able to add columns in snapshots.

Changes

  1. column_name > column.quoted (column_name does not exist, also in fabric)
  2. Changed into statement such that columns appear after SELECT *

@ericmuijsvanoord
Copy link
Contributor Author

@schlich Let me know If I need to do more :)

@schlich
Copy link
Contributor

schlich commented Feb 15, 2024

Thanks for the PR! Ideally I'd like to see a corresponding test that captures the bug -- The fix makes enough sense to me. Let's try to replicate this failure in a test:

I get an error when I add new columns to an existing snapshot (destination is available).

I'm not too familiar with snapshots so if you feel like you're not sure how to implement the test in the code base, let's discuss a high level approach here and figure out how it translates.

@ericmuijsvanoord
Copy link
Contributor Author

I don't have a full dev environment for this yet so might take a while.
High level approach:

  1. Create a source table
  2. Snapshot the table
  3. Add column to source table (varchar(100)) for example using ALTER TABLE source_table ADD COLUMN new_varchar varchar(100) NULL
  4. Snapshot again
  5. Validate if column is added tot the snapshot table

@ericmuijsvanoord
Copy link
Contributor Author

@schlich
Maybe we can just use the default dbt core tests, as these cover snapshots extensively.
It is true that by default the base tests are not run?

Ref: https://github.com/dbt-labs/dbt-core/tree/main/tests/functional/simple_snapshot

@cody-scott
Copy link
Collaborator

Adding my two cents.

Yes if there is a core test(s) from the default test suite then those should get implemented and overridden as needed.

@sdebruyn
Copy link
Member

sdebruyn commented Mar 4, 2024

Chiming in here: my colleagues and myself recently did a lot of work for dbt-synapse and dbt-fabric (bringing both to dbt 1.8). We also noticed that dbt-fabric does not implement all the tests it should. It has all tests until dbt 1.4 (that's the point until where I implemented all of them and transferred ownership to Microsoft) but all newer ones haven't been added yet. Our assignment focussed solely on dbt-synapse so we implemented them there. You might be able to leverage a lot of the work we did there.

@ericmuijsvanoord
Copy link
Contributor Author

Thanks for the comments @sdebruyn @cody-scott

See the latest commits in which I overwrite some of the snapshot test functions to get it to work well.

The main branch did not work as expected:

image

@ericmuijsvanoord
Copy link
Contributor Author

eagues and myself recently did a lot of work for dbt-synapse and dbt-fabric (bringing both to dbt 1.8). We also noticed that dbt-fabric does not implement all the tests it shou

Maybe a new issue / enhancement should be raised to solve this.

@ericmuijsvanoord
Copy link
Contributor Author

@schlich : let me know in case I need to update something

@schlich schlich merged commit f789ab0 into dbt-msft:master Mar 19, 2024
53 checks passed
@ericmuijsvanoord ericmuijsvanoord deleted the fix/add-columns branch March 25, 2024 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants