Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Better error message for drop statement when user does not have access to table/view #897

Open
2 tasks done
victor-frank-signet opened this issue Aug 17, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@victor-frank-signet
Copy link

Is this a new bug in dbt-redshift?

  • I believe this is a new bug in dbt-redshift
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When creating a view in redshift with jinja.

{{ 
	config(
    materialized = 'view',
	schema='test_schema'
    )
}}

with cte as (
    select * from testschema.test_table_a_tbl
)

select * from cte

The produced code is creating a a temp view and then sometimes running drop external if which is not valid redshift syntax:

  create view "db"."test_schema"."test_view_a_vw__dbt_tmp" as (
    

with cte as (
    select * from testschema.test_table_a_tbl
)

select * from cte
  ) ;
  
  drop external if exists "db"."test_schema"."test_view_a_vw" cascade

Expected Behavior

{{ 
	config(
    materialized = 'view',
	schema='test_schema'
    )
}}

with cte as (
    select * from testschema.test_table_a_tbl
)

select * from cte

Compiles to:

  create view "db"."test_schema"."test_view_a_vw__dbt_tmp" as (
    

with cte as (
    select * from testschema.test_table_a_tbl
)

select * from cte
  ) ;
  
  drop view if exists "db"."test_schema"."test_view_a_vw" cascade

Steps To Reproduce

  1. dbt-redshift 1.8.1 dbt-core==1.8.4|1.8.5
  2. dbt run --project-dir /tmp/dbt/test_project --profiles-dir /tmp/dbt/test_project --select testcomp.* --debug

Relevant log output

�[0m00:11:26  Began running node model.test_dir.test_view_a_vw
�[0m00:11:26  1 of 12 START sql view model test_schema_src.test_view_a_vw .... [RUN]
�[0m00:11:26  Re-using an available connection from the pool (formerly list_db_schema, now model.test_dir.test_view_a_vw)
�[0m00:11:26  Began compiling node model.test_dir.test_view_a_vw
�[0m00:11:26  Writing injected SQL for node "model.test_dir.test_view_a_vw"
�[0m00:11:26  Began executing node model.test_dir.test_view_a_vw
�[0m00:11:26  Writing runtime sql for node "model.test_dir.test_view_a_vw"
�[0m00:11:26  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:26  On model.test_dir.test_view_a_vw: BEGIN
�[0m00:11:26  Opening a new connection, currently in state closed
�[0m00:11:26  Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
�[0m00:11:26  Redshift adapter: Connecting to redshift with username/password based auth...
�[0m00:11:26  SQL status: SUCCESS in 0.064 seconds
�[0m00:11:26  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:26  On model.test_dir.test_view_a_vw: /* {"app": "dbt", "dbt_version": "1.8.4", "profile_name": "test_dir", "target_name": "env", "node_id": "model.test_dir.test_view_a_vw"} */


  create view "test_db"."test_schema_src"."test_view_a_vw__dbt_tmp" as (
    


with cte as (
    SELECT *
    FROM test_schema.test_schema_location_tbl ORG,

)

select 
    *
from cte
  ) ;
�[0m00:11:26  SQL status: SUCCESS in 0.033 seconds
�[0m00:11:26  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:26  On model.test_dir.test_view_a_vw: /* {"app": "dbt", "dbt_version": "1.8.4", "profile_name": "test_dir", "target_name": "env", "node_id": "model.test_dir.test_view_a_vw"} */
alter table "test_db"."test_schema_src"."test_view_a_vw" rename to "test_view_a_vw__dbt_backup"
�[0m00:11:26  SQL status: SUCCESS in 0.004 seconds
�[0m00:11:26  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:26  On model.test_dir.test_view_a_vw: /* {"app": "dbt", "dbt_version": "1.8.4", "profile_name": "test_dir", "target_name": "env", "node_id": "model.test_dir.test_view_a_vw"} */
alter table "test_db"."test_schema_src"."test_view_a_vw__dbt_tmp" rename to "test_view_a_vw"
�[0m00:11:26  SQL status: SUCCESS in 0.004 seconds
�[0m00:11:26  On model.test_dir.test_view_a_vw: COMMIT
�[0m00:11:26  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:26  On model.test_dir.test_view_a_vw: COMMIT
�[0m00:11:26  SQL status: SUCCESS in 0.460 seconds
�[0m00:11:26  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:26  On model.test_dir.test_view_a_vw: BEGIN
�[0m00:11:26  SQL status: SUCCESS in 0.003 seconds
�[0m00:11:26  Applying DROP to: "test_db"."test_schema_src"."test_view_a_vw__dbt_backup"
�[0m00:11:26  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:26  On model.test_dir.test_view_a_vw: /* {"app": "dbt", "dbt_version": "1.8.4", "profile_name": "test_dir", "target_name": "env", "node_id": "model.test_dir.test_view_a_vw"} */
drop view if exists "test_db"."test_schema_src"."test_view_a_vw__dbt_backup" cascade
�[0m00:11:27  SQL status: SUCCESS in 0.743 seconds
�[0m00:11:27  On model.test_dir.test_view_a_vw: COMMIT
�[0m00:11:27  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:27  On model.test_dir.test_view_a_vw: COMMIT
�[0m00:11:28  SQL status: SUCCESS in 0.472 seconds
�[0m00:11:28  Using redshift connection "model.test_dir.test_view_a_vw"
�[0m00:11:28  On model.test_dir.test_view_a_vw: BEGIN
�[0m00:11:28  SQL status: SUCCESS in 0.001 seconds
�[0m00:11:28  On model.test_dir.test_view_a_vw: ROLLBACK
�[0m00:11:28  On model.test_dir.test_view_a_vw: Close
�[0m00:11:28  Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '37206027-27cd-44a4-99ac-7e279f6c8dbd', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f1db1685650>]}
�[0m00:11:28  1 of 12 OK created sql view model test_schema_src.test_view_a_vw  [�[32mSUCCESS�[0m in 1.99s]
�[0m00:11:28  Finished running node model.test_dir.test_view_a_vw
�[0m00:11:28  Began running node model.test_dir.test_schema_time_dimension_vw
�[0m00:11:28  2 of 12 START sql view model test_schema_src.test_schema_time_dimension_vw ............. [RUN]
�[0m00:11:28  Re-using an available connection from the pool (formerly model.test_dir.test_view_a_vw, now model.test_dir.test_schema_time_dimension_vw)
�[0m00:11:28  Began compiling node model.test_dir.test_schema_time_dimension_vw
�[0m00:11:28  Writing injected SQL for node "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:28  Began executing node model.test_dir.test_schema_time_dimension_vw
�[0m00:11:28  Using redshift connection "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:28  On model.test_dir.test_schema_time_dimension_vw: BEGIN
�[0m00:11:28  Opening a new connection, currently in state closed
�[0m00:11:28  Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
�[0m00:11:28  Redshift adapter: Connecting to redshift with username/password based auth...
�[0m00:11:28  SQL status: SUCCESS in 0.081 seconds
�[0m00:11:28  Applying DROP to: "test_db"."test_schema_src"."test_schema_time_dimension_vw__dbt_tmp"
�[0m00:11:28  Using redshift connection "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:28  On model.test_dir.test_schema_time_dimension_vw: /* {"app": "dbt", "dbt_version": "1.8.4", "profile_name": "test_dir", "target_name": "env", "node_id": "model.test_dir.test_schema_time_dimension_vw"} */
drop view if exists "test_db"."test_schema_src"."test_schema_time_dimension_vw__dbt_tmp" cascade
�[0m00:11:28  SQL status: SUCCESS in 0.246 seconds
�[0m00:11:28  On model.test_dir.test_schema_time_dimension_vw: COMMIT
�[0m00:11:28  Using redshift connection "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:28  On model.test_dir.test_schema_time_dimension_vw: COMMIT
�[0m00:11:29  SQL status: SUCCESS in 1.450 seconds
�[0m00:11:29  Using redshift connection "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:29  On model.test_dir.test_schema_time_dimension_vw: BEGIN
�[0m00:11:29  SQL status: SUCCESS in 0.002 seconds
�[0m00:11:29  Writing runtime sql for node "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:29  Using redshift connection "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:29  On model.test_dir.test_schema_time_dimension_vw: /* {"app": "dbt", "dbt_version": "1.8.4", "profile_name": "test_dir", "target_name": "env", "node_id": "model.test_dir.test_schema_time_dimension_vw"} */


  create view "test_db"."test_schema_src"."test_schema_time_dimension_vw__dbt_tmp" as (
    

with cte as (
    select * from test_schema.test_schema_time_dimension_tbl
)

select * from cte
  ) ;
�[0m00:11:30  SQL status: SUCCESS in 0.039 seconds
�[0m00:11:30  On model.test_dir.test_schema_time_dimension_vw: COMMIT
�[0m00:11:30  Using redshift connection "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:30  On model.test_dir.test_schema_time_dimension_vw: COMMIT
�[0m00:11:30  SQL status: SUCCESS in 0.841 seconds
�[0m00:11:30  Using redshift connection "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:30  On model.test_dir.test_schema_time_dimension_vw: BEGIN
�[0m00:11:30  SQL status: SUCCESS in 0.008 seconds
�[0m00:11:30  Applying DROP to: "test_db"."test_schema_src"."test_schema_time_dimension_vw"
�[0m00:11:30  Using redshift connection "model.test_dir.test_schema_time_dimension_vw"
�[0m00:11:30  On model.test_dir.test_schema_time_dimension_vw: /* {"app": "dbt", "dbt_version": "1.8.4", "profile_name": "test_dir", "target_name": "env", "node_id": "model.test_dir.test_schema_time_dimension_vw"} */
drop external if exists "test_db"."test_schema_src"."test_schema_time_dimension_vw" cascade
�[0m00:11:30  Redshift adapter: Redshift error: syntax error at or near "if" in context "
drop external if", at line 2, column 15
�[0m00:11:30  On model.test_dir.test_schema_time_dimension_vw: ROLLBACK
�[0m00:11:30  Redshift adapter: Error running SQL: macro drop_relation
�[0m00:11:30  Redshift adapter: Rolling back transaction.
�[0m00:11:30  On model.test_dir.test_schema_time_dimension_vw: Close
�[0m00:11:30  Database Error in model test_schema_time_dimension_vw (models/comp/test_schema_time_dimension_vw.sql)
  syntax error at or near "if" in context "
  drop external if", at line 2, column 15
  compiled Code at target/run/test_dir/models/comp/test_schema_time_dimension_vw.sql
�[0m00:11:30  Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '37206027-27cd-44a4-99ac-7e279f6c8dbd', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f1db1685650>]}

Environment

- OS: Linux AMI
- Python: 3.9
- dbt-core: 1.8.4 or 1.8.5
- dbt-redshift: 1.8.1

Additional Context

No response

@victor-frank-signet victor-frank-signet added bug Something isn't working triage labels Aug 17, 2024
@victor-frank-signet victor-frank-signet changed the title [Bug] Some views are running drop external when are valid views starting Aug 9th [Bug] Some views are running drop external when are valid views Aug 20, 2024
@victor-frank-signet victor-frank-signet changed the title [Bug] Some views are running drop external when are valid views [Bug] Some views are running drop external when they are redshift internal views Aug 20, 2024
@victor-frank-signet
Copy link
Author

victor-frank-signet commented Aug 20, 2024

This is probably no longer a bug. This was resolved on my end by changing the permissions on the view and the table, then dbt could resolve they are not external. But I still think this at least an enhancement but maybe a bug because if the view exists but dbt user does not have permission it should not produce a invalid sql "drop external if " instead it should say permission error while running the correct redshift sql stament "drop view|table if"

It has something to do with how the relation.type is populated in: �[0m00:11:30 Redshift adapter: Error running SQL: macro drop_relation

@amychen1776 amychen1776 changed the title [Bug] Some views are running drop external when they are redshift internal views [Feature] Better error message for drop statement when user does not have access to table/view Aug 28, 2024
@amychen1776 amychen1776 added enhancement New feature or request and removed triage bug Something isn't working labels Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants