Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add macro to apply tags to source columns in on-run-end hook #16

Conversation

bsuryachandra
Copy link
Contributor

resolves #15

This is a:

  • documentation update
  • bug fix with no breaking changes
  • new functionality
  • a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

This PR introduces a new macro, apply_source_column_tags, which applies tags to the columns of all source tables. The macro can be used in the on-run-end hook to automate tagging source columns. It operates similarly to the existing apply_column_tags macro but is specifically designed for source tables.

The purpose of this change is to provide a streamlined method for applying tags to source columns, improving metadata management and enhancing data traceability. By tagging both models and source columns uniformly, it becomes easier to manage important or sensitive data within the data pipeline.

Checklist

  • This code is associated with an Issue which has been triaged and accepted for development
  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • Snowflake
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)

Copy link
Collaborator

@il-dat il-dat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bsuryachandra looks very great PR!

I have some comments from the general view, could you help to give it a look? I will try to dive into testing this PR in they next days, will approve it if all looks good 🏃‍♂️

macros/apply-tags/apply_tags.sql Outdated Show resolved Hide resolved
macros/apply-tags/apply_source_column_tags.yml Outdated Show resolved Hide resolved
Copy link
Collaborator

@thu-IL thu-IL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please also modify the model verify_if_masking_policies_applied_correctly to check if tags are applied correctly to the source columns as intended?

pyproject.toml Outdated Show resolved Hide resolved
@il-dat il-dat merged commit a693325 into infinitelambda:main Nov 11, 2024
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Applying Tags on columns in source table
3 participants