Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement: include sourcename or modelname in posthook to minimise overhead on run model tasks #38

Open
Chobsz opened this issue Jun 9, 2022 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@Chobsz
Copy link

Chobsz commented Jun 9, 2022

#41

I've edited the code to allow for the following post hooks in the project file to process only the relevant maskingpolicies on a run task.

                +post-hook: "{{ dbt_snow_mask.apply_masking_policy('models','mymodelname')}}" 
                +post-hook: "{{ dbt_snow_mask.apply_masking_policy('sources','mysourcename')}}" 

-- Altered macro: [apply_masking_policy] (50% the old macro)

{% macro apply_masking_policy(resource_type="models",resource_name="undefined",meta_key="masking_policy") %}

    {% if execute %}

        {% if resource_type == "sources" and  resource_name == "undefined" %}
            {{ dbt_snow_mask.apply_masking_policy_list_for_sources(meta_key) }}
        {% elif resource_type == "models" and resource_name == "undefined" %}
            {{ dbt_snow_mask.apply_masking_policy_list_for_models(meta_key) }}           
        {% elif resource_type == "sources" and resource_name != "undefined" %}
            {{ dbt_snow_mask.apply_masking_policy_list_for_onesource(meta_key,resource_name) }}
        {% elif resource_type == "models" and resource_name != "undefined" %}
            {{ dbt_snow_mask.apply_masking_policy_list_for_onemodel(meta_key,resource_name) }}
        {% endif %}
    {% endif %}

{% endmacro %}

-- New macro: [apply_masking_policy_list_for_onesource] (95% the old macro apply_masking_policy_list_for_sources)
--Most important change:

% macro apply_masking_policy_list_for_onesource(meta_key,resource_name,operation_type="apply") %}

    {% for node in graph.sources.values() -%}
        {%- if node.source_name == resource_name -%}
 [...]

-- New macro: [apply_masking_policy_list_for_onemodel] (95% the old macro apply_masking_policy_list_for_models)
--Most important change:

{% macro apply_masking_policy_list_for_onemodel(meta_key,resource_name,operation_type="apply") %}
{%set resource_name =  '/'+ resource_name +'/' %}
    {% if operation_type == "apply" %}    
        {% for node in graph.nodes.values() %}
            {%- if resource_name in node.path  -%}          
 [...]

If you need the whole code, I created a PullRequest

@Chobsz Chobsz changed the title Improvement: include sourcename of model name in posthook to minimise overhead on run model tasks Improvement: include sourcename or modelname in posthook to minimise overhead on run model tasks Jun 9, 2022
@entechlog entechlog assigned entechlog and Chobsz and unassigned entechlog Jun 11, 2022
@entechlog entechlog added the enhancement New feature or request label Jun 11, 2022
@Chobsz
Copy link
Author

Chobsz commented Jun 13, 2022

created PR 41, let me know if you need anything.

@Chobsz
Copy link
Author

Chobsz commented Jan 13, 2023

created PR #58, hope it meets standards.
however the whole thing could have been done wth DRY, but my Jinja is just nog up to par.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants