Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add update_non_tracked_columns flag for snapshot materialization #10933

Open
3 tasks done
talpontera opened this issue Oct 29, 2024 · 0 comments
Open
3 tasks done

Add update_non_tracked_columns flag for snapshot materialization #10933

talpontera opened this issue Oct 29, 2024 · 0 comments
Labels
enhancement New feature or request triage

Comments

@talpontera
Copy link

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Add a new configuration flag update_non_tracked_columns to dbt's snapshot materialization that allows immediate updates for non-tracked columns while maintaining SCD Type 2 behavior for tracked columns. When enabled, this flag would automatically generate the necessary post-hook to update non-tracked columns.

post_hook="update {{ this }} target set
{% for col in non_tracked_columns %}
{{ col.name }} = source.{{ col.name }},
{% endfor %}
dbt_updated_at = current_timestamp()
from source
where target.unique_key = source.unique_key
and columns_changed"

Describe alternatives you've considered

Current approach: Manually writing post-hooks (more verbose, error-prone)

Who will this benefit?

Teams using snapshot tables where only specific columns need historical tracking
Users who want to avoid writing complex post-hooks manually
Projects migrating from other data warehousing solutions with similar hybrid update patterns

Example use case: Organization hierarchy tables where structural changes (org relationships, roles) need versioning but metadata (contact info, settings) should always be current.

Are you interested in contributing this feature?

Teams using snapshot tables where only specific columns need historical tracking Users who want to avoid writing complex post-hooks manually Projects migrating from other data warehousing solutions with similar hybrid update patterns Example use case: Organization hierarchy tables where structural changes (org relationships, roles) need versioning but metadata (contact info, settings) should always be current.

Anything else?

This feature would build on existing snapshot functionality and simply automate what users are currently doing manually with post-hooks. The implementation is straightforward and maintains all existing snapshot behaviors while adding convenience for a common use case.

@talpontera talpontera added enhancement New feature or request triage labels Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request triage
Projects
None yet
Development

No branches or pull requests

1 participant