Block taking jinja2.runtime.Undefined into DatabricksAdapter #98

ueshin · 2022-05-23T18:09:49Z

Description

Blocks taking jinja2.runtime.Undefined into DatabricksAdapter.

When calling the adapter method from macros, jinja2.runtime.Undefined can be passed to DatabricksAdapter that could cause unexpected behavior.

For example,

dbt-databricks/dbt/include/databricks/macros/materializations/snapshot.sql

Line 33 in b8152b6

{% if not adapter.check_schema_exists(model.database, model.schema) %}

The model.database is not defined and jinja2.runtime.Undefined will be used to call adapter.check_schema_exists and Cursor.schema() will fail:

[debug] [Thread-13 ]: Databricks adapter: Error while running:
GetSchemas(database=, schema=test16533281379208408401_test_basic)
[debug] [Thread-13 ]: Databricks adapter: <class 'databricks.sql.exc.RequestError'>: Error during request to server
[debug] [Thread-13 ]: finished collecting timing info
[debug] [Thread-13 ]: On snapshot.snapshot_strategy_timestamp.ts_snapshot: Close
[error] [Thread-13 ]: Unhandled error while executing snapshot.snapshot_strategy_timestamp.ts_snapshot
Error during request to server
  Unhandled error while executing snapshot.snapshot_strategy_timestamp.ts_snapshot
Error during request to server

allisonwang-db · 2022-05-23T19:26:55Z

If we call adapter.check_schema_exists(model.database, model.schema) with model.database = jinja2.runtime.Undefined, what will happen?

ueshin · 2022-05-23T19:35:11Z

@allisonwang-db

If we call adapter.check_schema_exists(model.database, model.schema) with model.database = jinja2.runtime.Undefined, what will happen?

For example, cursor.schema() call will fail:

[debug] [Thread-13 ]: Databricks adapter: Error while running:
GetSchemas(database=, schema=test16533281379208408401_test_basic)
[debug] [Thread-13 ]: Databricks adapter: <class 'databricks.sql.exc.RequestError'>: Error during request to server
[debug] [Thread-13 ]: finished collecting timing info
[debug] [Thread-13 ]: On snapshot.snapshot_strategy_timestamp.ts_snapshot: Close
[error] [Thread-13 ]: Unhandled error while executing snapshot.snapshot_strategy_timestamp.ts_snapshot
Error during request to server
  Unhandled error while executing snapshot.snapshot_strategy_timestamp.ts_snapshot
Error during request to server

Also updated the description.

allisonwang-db · 2022-05-24T01:20:09Z

dbt/adapters/databricks/relation.py

+        if "database" not in data["path"]:
+            data["path"]["database"] = None
+        else:
+            data["path"]["database"] = remove_undefined(data["path"]["database"])


I am curious when will "database" here be Undefined? Is there a specific configuration that can trigger this?

While parsing profiles, models, etc. and building metadata that happen before setting up DatabricksRelation, it could contain Undefined.

Also macros for snapshot seems to cause the issue after we change DatabricksRelation.include_policy.database=True.

Do you think it might be helpful to add a test case here (if we set DatabricksRelation.include_policy.database=True)

I'm not sure what kind of test in your mind?

For example, construct a test with the database field being jinja Undefined.

Added a test. Thanks!

allisonwang-db

Looks good! Thanks for adding the test.

ueshin · 2022-05-25T23:48:48Z

Thanks! merging.

…101) ### Description Backport of #98 Blocks taking `jinja2.runtime.Undefined` into `DatabricksAdapter`. When calling the adapter method from macros, `jinja2.runtime.Undefined` can be passed to `DatabricksAdapter` that could cause unexpected behavior. For example, https://github.com/databricks/dbt-databricks/blob/b8152b697b003e02fb52e86f59b2bde4b492cc0a/dbt/include/databricks/macros/materializations/snapshot.sql#L33 The `model.database` is not defined and `jinja2.runtime.Undefined` will be used to call `adapter.check_schema_exists` and `Cursor.schema()` will fail: ``` [debug] [Thread-13 ]: Databricks adapter: Error while running: GetSchemas(database=, schema=test16533281379208408401_test_basic) [debug] [Thread-13 ]: Databricks adapter: <class 'databricks.sql.exc.RequestError'>: Error during request to server [debug] [Thread-13 ]: finished collecting timing info [debug] [Thread-13 ]: On snapshot.snapshot_strategy_timestamp.ts_snapshot: Close [error] [Thread-13 ]: Unhandled error while executing snapshot.snapshot_strategy_timestamp.ts_snapshot Error during request to server Unhandled error while executing snapshot.snapshot_strategy_timestamp.ts_snapshot Error during request to server ```

) ### Description Backport of #98 Blocks taking `jinja2.runtime.Undefined` into `DatabricksAdapter`. When calling the adapter method from macros, `jinja2.runtime.Undefined` can be passed to `DatabricksAdapter` that could cause unexpected behavior. For example, https://github.com/databricks/dbt-databricks/blob/b8152b697b003e02fb52e86f59b2bde4b492cc0a/dbt/include/databricks/macros/materializations/snapshot.sql#L33 The `model.database` is not defined and `jinja2.runtime.Undefined` will be used to call `adapter.check_schema_exists` and `Cursor.schema()` will fail: ``` [debug] [Thread-13 ]: Databricks adapter: Error while running: GetSchemas(database=, schema=test16533281379208408401_test_basic) [debug] [Thread-13 ]: Databricks adapter: <class 'databricks.sql.exc.RequestError'>: Error during request to server [debug] [Thread-13 ]: finished collecting timing info [debug] [Thread-13 ]: On snapshot.snapshot_strategy_timestamp.ts_snapshot: Close [error] [Thread-13 ]: Unhandled error while executing snapshot.snapshot_strategy_timestamp.ts_snapshot Error during request to server Unhandled error while executing snapshot.snapshot_strategy_timestamp.ts_snapshot Error during request to server ```

Block taking jinja2.runtime.Undefined into DatabricksAdapter.

b8152b6

ueshin requested a review from allisonwang-db May 23, 2022 18:09

ueshin mentioned this pull request May 23, 2022

[1.0] Block taking jinja2.runtime.Undefined into DatabricksAdapter. #99

Merged

changelog

6841705

ueshin mentioned this pull request May 23, 2022

Use 3-level namespace when catalog is set. #94

Merged

This was referenced May 23, 2022

Avoid using Cursor.schema API when database is None #100

Merged

[1.1] Block taking jinja2.runtime.Undefined into DatabricksAdapter. #101

Merged

allisonwang-db approved these changes May 24, 2022

View reviewed changes

ueshin added 2 commits May 24, 2022 13:51

Merge branch 'main' into undefined_proof

9dea5f9

Test

a7f77e8

ueshin requested a review from allisonwang-db May 25, 2022 17:37

allisonwang-db approved these changes May 25, 2022

View reviewed changes

ueshin merged commit 6606aa6 into databricks:main May 25, 2022

ueshin deleted the undefined_proof branch May 25, 2022 23:49

dataders mentioned this pull request Oct 3, 2022

Input from dbt Labs's core adapters team #201

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Block taking jinja2.runtime.Undefined into DatabricksAdapter #98

Block taking jinja2.runtime.Undefined into DatabricksAdapter #98

ueshin commented May 23, 2022 •

edited

Loading

allisonwang-db commented May 23, 2022

ueshin commented May 23, 2022 •

edited

Loading

allisonwang-db May 24, 2022

ueshin May 24, 2022

ueshin May 24, 2022 •

edited

Loading

allisonwang-db May 24, 2022

ueshin May 24, 2022

allisonwang-db May 25, 2022

ueshin May 25, 2022

allisonwang-db left a comment

ueshin commented May 25, 2022

Block taking jinja2.runtime.Undefined into DatabricksAdapter #98

Block taking jinja2.runtime.Undefined into DatabricksAdapter #98

Conversation

ueshin commented May 23, 2022 • edited Loading

Description

allisonwang-db commented May 23, 2022

ueshin commented May 23, 2022 • edited Loading

allisonwang-db May 24, 2022

Choose a reason for hiding this comment

ueshin May 24, 2022

Choose a reason for hiding this comment

ueshin May 24, 2022 • edited Loading

Choose a reason for hiding this comment

allisonwang-db May 24, 2022

Choose a reason for hiding this comment

ueshin May 24, 2022

Choose a reason for hiding this comment

allisonwang-db May 25, 2022

Choose a reason for hiding this comment

ueshin May 25, 2022

Choose a reason for hiding this comment

allisonwang-db left a comment

Choose a reason for hiding this comment

ueshin commented May 25, 2022

ueshin commented May 23, 2022 •

edited

Loading

ueshin commented May 23, 2022 •

edited

Loading

ueshin May 24, 2022 •

edited

Loading