Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Grant SQL Macros #5369

Merged
merged 37 commits into from
Jul 11, 2022
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
6e88fe7
init push or ct-660 work
McKnight-42 Jun 13, 2022
6d4b938
changes to default versions of get_show_grant_sql and get_grant_sql
McKnight-42 Jun 14, 2022
ea35fd1
completing init default versions of all macros being called for look …
McKnight-42 Jun 14, 2022
7e2b707
minor update to should_revoke
McKnight-42 Jun 14, 2022
898aa8f
post pairing push up (does have log statements to make sure we remove)
McKnight-42 Jun 15, 2022
76050da
minor spacing changes
McKnight-42 Jun 15, 2022
7a3e7c6
Merge branch 'main' of github.com:dbt-labs/dbt into ct-660-grant-sql
McKnight-42 Jun 16, 2022
5fb07c1
minor changes, and removal of logs so people can have clean grab of code
McKnight-42 Jun 16, 2022
4cf705f
minor changes to how get_revoke_sql works
McKnight-42 Jun 17, 2022
2fff84b
init attempt at applying apply_grants to all materialzations
McKnight-42 Jun 17, 2022
6fccef6
name change from recipents -> grantee
McKnight-42 Jun 17, 2022
528dff6
minor changes
McKnight-42 Jun 17, 2022
95f7ae4
Merge branch 'main' of github.com:dbt-labs/dbt into ct-660-grant-sql
McKnight-42 Jun 21, 2022
9079c4a
working on making a context to handle the diff gathering between gran…
McKnight-42 Jun 21, 2022
6bdb4b5
removing logs from most materializations to better track diff of gran…
McKnight-42 Jun 22, 2022
597ab05
starting to build out postgres get_show_grant_sql getting empty query…
McKnight-42 Jun 22, 2022
138f443
6/27 eod update looking into diff_grants variable not getting passed …
McKnight-42 Jun 27, 2022
fc7e24b
changes to loop cases
McKnight-42 Jun 28, 2022
6f53b3d
changes after pairing meeting
McKnight-42 Jun 28, 2022
b3e37cb
Merge branch 'main' of github.com:dbt-labs/dbt into ct-660-grant-sql
McKnight-42 Jun 29, 2022
99b1445
adding apply_grants to create_or_replace_view.sql
McKnight-42 Jul 5, 2022
7946a3b
models are building but testing out small issues around revoke statem…
McKnight-42 Jul 5, 2022
0597398
postgrest must fixes from jeremy's feedback
McKnight-42 Jul 6, 2022
1d263b7
postgres minor change to standarize_grants_dict
McKnight-42 Jul 6, 2022
c2e9aeb
updating after pairing with dough and jeremey incorporating the new v…
McKnight-42 Jul 6, 2022
0843f66
Merge branch 'main' of github.com:dbt-labs/dbt into ct-660-grant-sql
McKnight-42 Jul 6, 2022
077e4ff
adding ref of diff_of_two_dicts to base keys ref
McKnight-42 Jul 6, 2022
ab6be85
change of method type for standardize_grants_dict
McKnight-42 Jul 6, 2022
da557d9
minor update trying to fix unit test
McKnight-42 Jul 6, 2022
f2c957f
changes based on morning feedback
McKnight-42 Jul 7, 2022
eb935d5
change log message in default_apply_grants macro
McKnight-42 Jul 7, 2022
cefcd4f
CT-808 grant adapter tests (#5447)
gshank Jul 7, 2022
bdc0c71
rename grant[privilege] -> grant_config[privilege]
McKnight-42 Jul 7, 2022
67f5beb
Merge branch 'main' of github.com:dbt-labs/dbt into ct-660-grant-sql
McKnight-42 Jul 11, 2022
72bdae9
postgres macro rename to copy_grants
McKnight-42 Jul 11, 2022
794f4d1
CT-808 more grant adapter tests (#5452)
gshank Jul 11, 2022
068c59b
update to main, create changelog, whitespace fixes
McKnight-42 Jul 11, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
{% macro get_show_grant_sql(relation) %}
{{ return(adapter.dispatch("get_show_grant_sql", "dbt")(relation)) }}
{% endmacro %}

{% macro default__get_show_grant_sql(relation) %}
show grants on {{ relation.type }} {{ relation }}
{% endmacro %}
jtcohen6 marked this conversation as resolved.
Show resolved Hide resolved

{% macro get_grant_sql(relation, grant_config) %}
{{ return(adapter.dispatch('get_grant_sql', 'dbt')(relation, grant_config)) }}
{% endmacro %}

{% macro default__get_grant_sql(relation, grant_config) %}
McKnight-42 marked this conversation as resolved.
Show resolved Hide resolved
{% for privilege in grant_config.keys() %}
{% set grantee = grant_config[privilege] %}
grant {{ privilege }} on {{ relation.type }} {{ relation }} to {{ grantee | join(', ') }}
{% endfor %}
{% endmacro %}

{% macro get_revoke_sql(relation, grant_config) %}
{{ return(adapter.dispatch("get_revoke_sql", "dbt")(relation, grant_config)) }}
{% endmacro %}

{% macro default__get_revoke_sql(relation, grant_config) %}
{% for privilege in grant_config.keys() %}
{% set grantee = grant_config[privilege] %}
revoke {{ privilege }} on {{ relation.type }} {{ relation }} from {{ grantee | join(', ') }}
where {{ grantee }} != {{ target.user }}
{% endfor %}
{% endmacro %}

{% macro apply_grants(relation, grant_config, should_revoke) %}
{{ return(adapter.dispatch("apply_grants", "dbt")(relation, grant_config, should_revoke)) }}
{% endmacro %}

{% macro default__apply_grants(relation, grant_config, should_revoke=True) %}
{% if grant_config %}
jtcohen6 marked this conversation as resolved.
Show resolved Hide resolved
{% call statement('grants') %}
{% if should_revoke %}
{% set current_grants = run_query(get_show_grant_sql(relation)) %}
{{ log(current_grants, info = true) }}
{% set diff_grants = { k: current_grants[k] for k in set(current_grants) - set(grant_config) } %}
jtcohen6 marked this conversation as resolved.
Show resolved Hide resolved

{{ log(diff_grants, info = true) }}
{{ get_revoke_sql(relation, diff_grants) }}
{% endif %}
{{ get_grant_sql(relation, grant_config) }}
{% endcall %}
{% endif %}
{% endmacro %}
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
-- BEGIN, in a separate transaction
{%- set preexisting_intermediate_relation = load_cached_relation(intermediate_relation)-%}
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
-- grab current tables grants config for comparision later on
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not something needed to do before merging, but we added the same change pretty much for all materialization methods. Should we consider refactoring the code so we can only do it in one place?

{% set grant_config = config.get('grants') %}
McKnight-42 marked this conversation as resolved.
Show resolved Hide resolved
{{ drop_relation_if_exists(preexisting_intermediate_relation) }}
{{ drop_relation_if_exists(preexisting_backup_relation) }}

Expand Down Expand Up @@ -59,6 +61,9 @@
{% do to_drop.append(backup_relation) %}
{% endif %}

{{ log(grant_config, "what grants are we passing") }}
{% do apply_grants(target_relation, grant_config) %}

{% do persist_docs(target_relation, model) %}

{% if existing_relation is none or existing_relation.is_view or should_full_refresh() %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
{%- set backup_relation = make_backup_relation(target_relation, backup_relation_type) -%}
-- as above, the backup_relation should not already exist
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
-- grab current tables grants config for comparision later on
{% set grant_config = config.get('grants') %}
McKnight-42 marked this conversation as resolved.
Show resolved Hide resolved

-- drop the temp relations if they exist already in the database
{{ drop_relation_if_exists(preexisting_intermediate_relation) }}
Expand All @@ -39,7 +41,8 @@
{% do create_indexes(target_relation) %}

{{ run_hooks(post_hooks, inside_transaction=True) }}

{{ log(grant_config, "what grants are we passing") }}
{% do apply_grants(target_relation, grant_config) %}
{% do persist_docs(target_relation, model) %}

-- `COMMIT` happens here
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
{%- set backup_relation = make_backup_relation(target_relation, backup_relation_type) -%}
-- as above, the backup_relation should not already exist
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
-- grab current tables grants config for comparision later on
{% set grant_config = config.get('grants') %}
McKnight-42 marked this conversation as resolved.
Show resolved Hide resolved

{{ run_hooks(pre_hooks, inside_transaction=False) }}

Expand All @@ -47,6 +49,9 @@
{% endif %}
{{ adapter.rename_relation(intermediate_relation, target_relation) }}

{{ log(grant_config, "what grants are we passing") }}
{% do apply_grants(target_relation, grant_config) %}

{% do persist_docs(target_relation, model) %}

{{ run_hooks(post_hooks, inside_transaction=True) }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
{%- set exists_as_view = (old_relation is not none and old_relation.is_view) -%}

{%- set agate_table = load_agate_table() -%}
-- grab current tables grants config for comparision later on
{% set grant_config = config.get('grants') %}

{%- do store_result('agate_table', response='OK', agate_table=agate_table) -%}

{{ run_hooks(pre_hooks, inside_transaction=False) }}
Expand All @@ -35,6 +38,8 @@
{% endcall %}

{% set target_relation = this.incorporate(type='table') %}
{{ log(grant_config, "what grants are we passing") }}
{% do apply_grants(target_relation, grant_config) %}
{% do persist_docs(target_relation, model) %}

{% if full_refresh_mode or not exists_as_table %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@

{%- set strategy_name = config.get('strategy') -%}
{%- set unique_key = config.get('unique_key') %}
-- grab current tables grants config for comparision later on
{% set grant_config = config.get('grants') %}

{% set target_relation_exists, target_relation = get_or_create_relation(
database=model.database,
Expand Down Expand Up @@ -72,7 +74,8 @@
{% call statement('main') %}
{{ final_sql }}
{% endcall %}

{{ log(grant_config, "what grants are we passing") }}
{% do apply_grants(target_relation, grant_config) %}
{% do persist_docs(target_relation, model) %}

{% if not target_relation_exists %}
Expand Down