Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

model contracts + constraints with nested fields #738

Merged
merged 16 commits into from
Jun 14, 2023

Conversation

MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented May 24, 2023

resolves #673

Description

🎩

-- nested_fields.yml
version: 2

models:
  - name: double_nested_fields
    config:
      contract: {enforced: true}
      materialized: table
    columns:
      - name: a
        data_type: string
      - name: b.id
        data_type: integer
        constraints:
          - type: not_null
      - name: b.name
        description: test description
        data_type: string
      - name: b.double_nested.id
        data_type: integer
      - name: b.double_nested.another.again
        data_type: string
      - name: b.double_nested.another.even_more
        data_type: integer
        constraints: 
          - type: not_null
-- double_nested_fields.sql
select
  'string' as a,
  struct(
    1 as id,
    'name' as name,
    struct(2 as id, struct('test' as again, '2' as even_more) as another) as double_nested
  ) as b
❯ dbt run --select double_nested_fields --project-dir ~/basic-dbt
...
21:31:07  
21:31:07  Compilation Error in model double_nested_fields (models/double_nested_fields.sql)
21:31:07    This model has an enforced contract that failed.
21:31:07    Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.
21:31:07    
21:31:07    | column_name                       | definition_type | contract_type | mismatch_reason    |
21:31:07    | --------------------------------- | --------------- | ------------- | ------------------ |
21:31:07    | b.double_nested.another.even_more | STRING          | INT64         | data type mismatch |

Fixing the incorrect data type, dbt run passes ✅

Checklist

@cla-bot cla-bot bot added the cla:yes label May 24, 2023
@github-actions
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-bigquery contributing guide.

1 similar comment
@github-actions
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-bigquery contributing guide.

@MichelleArk MichelleArk changed the title first pass: model contracts with nested fields model contracts + constraints with nested fields May 26, 2023
@MichelleArk
Copy link
Contributor Author

Opened a follow-up tech debt issue to adopt this convention + implementation in dbt-core at some point in the future: dbt-labs/dbt-core#7759. Presently, BigQuery's support of nested columns + constraints is most maturely supported and keeping the implementation focused on BigQuery will prevent a large scope increase of this issue.

@MichelleArk MichelleArk marked this pull request as ready for review June 1, 2023 22:08
@MichelleArk MichelleArk requested a review from a team as a code owner June 1, 2023 22:08
Copy link
Contributor

@gshank gshank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code itself looks fine (though I'm not completely familiar). There's a failing functional test which might be related though.

@MichelleArk
Copy link
Contributor Author

The failing integration test looks related to some recently merged changes for fixes to foreign key rendering that made the base test a little less generic than it previously was. I have a fix in another PR related to constraints but will move it out into its own PR to unblock this work.

@MichelleArk MichelleArk merged commit b1e950b into main Jun 14, 2023
@MichelleArk MichelleArk deleted the arky/nested-columns-contract branch June 14, 2023 17:34
@github-actions
Copy link
Contributor

The backport to 1.5.latest failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.5.latest 1.5.latest
# Navigate to the new working tree
cd .worktrees/backport-1.5.latest
# Create a new branch
git switch --create backport-738-to-1.5.latest
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b1e950bf6f8f6c3528b65080e257b746839ef87c
# Push it to GitHub
git push --set-upstream origin backport-738-to-1.5.latest
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.5.latest

Then, create a pull request where the base branch is 1.5.latest and the compare/head branch is backport-738-to-1.5.latest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[ADAP-465] [Bug] Breaking schema changes for nested fields
3 participants