Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Model level descriptions aren't being added to Python models #740

Closed
jeremyyeo opened this issue Jul 24, 2024 · 1 comment · Fixed by #743
Closed

Model level descriptions aren't being added to Python models #740

jeremyyeo opened this issue Jul 24, 2024 · 1 comment · Fixed by #743
Labels
bug Something isn't working

Comments

@jeremyyeo
Copy link

Describe the bug

As per title - Python models aren't getting model level descriptions added to them.

Steps To Reproduce

Project setup.

# dbt_project.yml
name: my_dbt_project
profile: all
config-version: 2
version: "1.0.0"

models:
   my_dbt_project:
      +materialized: table
      +persist_docs:
         relation: true
         columns: true
# models/schema.yml
models:
  - name: my_sql_model
    description: Model description.
    columns:
      - name: id
        description: Column description.
  - name: my_python_model
    description: Model description.
    columns:
      - name: id
        description: Column description.
-- models/my_sql_model.sql
select 1 id
# models/my_python_model.py
import pandas as pd
def model(dbt, session):
    dbt.config(
        submission_method="all_purpose_cluster",
        cluster_id="0426-154059-dox4kjfg"
    )
    return pd.DataFrame({"id": [1]})

Build everything:

$ dbt --debug run

01:12:31  Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1143f57d0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x114439290>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x114446d90>]}
01:12:31  Running with dbt=1.8.4
01:12:31  running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/Users/jeremy/.dbt', 'fail_fast': 'False', 'debug': 'True', 'log_path': '/Users/jeremy/git/dbt-basic/logs', 'version_check': 'True', 'warn_error': 'None', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'empty': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'invocation_command': 'dbt --debug run', 'introspect': 'True', 'target_path': 'None', 'log_format': 'default', 'send_anonymous_usage_stats': 'True'}
01:12:31  Spark adapter: Setting pyhive.hive logging to ERROR
01:12:31  Spark adapter: Setting thrift.transport logging to ERROR
01:12:31  Spark adapter: Setting thrift.protocol logging to ERROR
01:12:32  Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '88350355-ab6d-4ace-8c7f-b78c15812d11', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x16affa3d0>]}
01:12:32  Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '88350355-ab6d-4ace-8c7f-b78c15812d11', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x16aabd650>]}
01:12:32  Registered adapter: databricks=1.8.4
01:12:32  checksum: 54398663eddc8e4ac172fdab397e09b2aa6984947a84e6ec7983096c3fae7b7a, vars: {}, profile: , target: , version: 1.8.4
01:12:32  Partial parsing enabled: 0 files deleted, 0 files added, 2 files changed.
01:12:32  Partial parsing: updated file: my_dbt_project://models/my_sql_model.sql
01:12:32  Partial parsing: updated file: my_dbt_project://models/my_python_model.py
01:12:32  Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '88350355-ab6d-4ace-8c7f-b78c15812d11', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x16aa8b8d0>]}
01:12:32  Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '88350355-ab6d-4ace-8c7f-b78c15812d11', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x16c06b590>]}
01:12:32  Found 2 models, 585 macros
01:12:32  Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '88350355-ab6d-4ace-8c7f-b78c15812d11', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1143dffd0>]}
01:12:32  
01:12:32  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=0s, acquire-count=0, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Creating connection
01:12:32  Acquiring new databricks connection 'master'
01:12:32  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=1.1920928955078125e-06s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Acquired connection on thread (44871, 8670427840), using default compute resource
01:12:32  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=None, name=list_hive_metastore, idle-time=0s, acquire-count=0, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Creating connection
01:12:32  Acquiring new databricks connection 'list_hive_metastore'
01:12:32  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=None, name=list_hive_metastore, idle-time=2.1457672119140625e-06s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Acquired connection on thread (44871, 6235451392), using default compute resource
01:12:32  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=None, name=list_hive_metastore, idle-time=0.00018405914306640625s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:32  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=None, name=list_hive_metastore, idle-time=0.0004150867462158203s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:12:32  Using databricks connection "list_hive_metastore"
01:12:32  On list_hive_metastore: GetSchemas(database=hive_metastore, schema=None)
01:12:32  Opening a new connection, currently in state init
01:12:34  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore, idle-time=6.9141387939453125e-06s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Connection created
01:12:34  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:12:35  SQL status: OK in 2.2799999713897705 seconds
01:12:35  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-cf4e-18ea-ba67-1f6f2413ba6a) - Closing cursor
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore, idle-time=5.0067901611328125e-06s, acquire-count=0, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Released connection
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore, idle-time=0.000614166259765625s, acquire-count=0, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:35  Re-using an available connection from the pool (formerly list_hive_metastore, now list_hive_metastore_dbt_jyeo)
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.00115203857421875s, acquire-count=0, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Reusing connection previously named list_hive_metastore
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.0013339519500732422s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Acquired connection on thread (44871, 6235451392), using default compute resource
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.001516103744506836s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.001811981201171875s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:12:35  Using databricks connection "list_hive_metastore_dbt_jyeo"
01:12:35  On list_hive_metastore_dbt_jyeo: GetTables(database=hive_metastore, schema=dbt_jyeo, identifier=None)
01:12:35  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:12:35  SQL status: OK in 0.7400000095367432 seconds
01:12:35  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-cf9c-1a5c-85e8-2c23859e85ac) - Closing cursor
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.7537238597869873s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.7540349960327148s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.7542510032653809s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:35  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.7544209957122803s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:12:35  Spark adapter: NotImplemented: add_begin_query
01:12:35  Using databricks connection "list_hive_metastore_dbt_jyeo"
01:12:35  On list_hive_metastore_dbt_jyeo: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "connection_name": "list_hive_metastore_dbt_jyeo"} */

      select current_catalog()
  
01:12:35  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:12:36  SQL status: OK in 0.41999998688697815 seconds
01:12:36  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-d002-1c24-858f-fe1253655bef) - Closing cursor
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=1.1875240802764893s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=1.1878960132598877s, acquire-count=1, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:12:36  Using databricks connection "list_hive_metastore_dbt_jyeo"
01:12:36  On list_hive_metastore_dbt_jyeo: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "connection_name": "list_hive_metastore_dbt_jyeo"} */
show views in `hive_metastore`.`dbt_jyeo`
  
01:12:36  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:12:36  SQL status: OK in 0.5299999713897705 seconds
01:12:36  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-d04d-174e-b2bf-5f2fc19dd746) - Closing cursor
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=5.0067901611328125e-06s, acquire-count=0, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Released connection
01:12:36  Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '88350355-ab6d-4ace-8c7f-b78c15812d11', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x16c1e90d0>]}
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=4.038593053817749s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Checking idleness
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=4.0387961864471436s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Retrieving connection
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=4.038973331451416s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Checking idleness
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=4.039152145385742s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Retrieving connection
01:12:36  Spark adapter: NotImplemented: add_begin_query
01:12:36  Spark adapter: NotImplemented: commit
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=9.5367431640625e-07s, acquire-count=0, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Released connection
01:12:36  Concurrency: 1 threads (target='db')
01:12:36  
01:12:36  Began running node model.my_dbt_project.my_python_model
01:12:36  1 of 2 START python table model dbt_jyeo.my_python_model ....................... [RUN]
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=list_hive_metastore_dbt_jyeo, idle-time=0.02280902862548828s, acquire-count=0, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:36  Re-using an available connection from the pool (formerly list_hive_metastore_dbt_jyeo, now model.my_dbt_project.my_python_model)
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=0.023138999938964844s, acquire-count=0, language=None, thread-identifier=(44871, 6235451392), compute-name=) - Reusing connection previously named list_hive_metastore_dbt_jyeo
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=0.023319005966186523s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Acquired connection on thread (44871, 6235451392), using default compute resource for model '`hive_metastore`.`dbt_jyeo`.`my_python_model`'
01:12:36  Began compiling node model.my_dbt_project.my_python_model
01:12:36  Writing injected SQL for node "model.my_dbt_project.my_python_model"
01:12:36  Began executing node model.my_dbt_project.my_python_model
01:12:36  MATERIALIZING TABLE
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=0.048874855041503906s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:36  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=0.049345970153808594s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:12:36  Using databricks connection "model.my_dbt_project.my_python_model"
01:12:36  On model.my_dbt_project.my_python_model: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "node_id": "model.my_dbt_project.my_python_model"} */

      describe extended `hive_metastore`.`dbt_jyeo`.`my_python_model`
  
01:12:36  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:12:37  SQL status: OK in 0.6200000047683716 seconds
01:12:37  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-d0ad-1005-93d1-3ff7d3ef0158) - Closing cursor
01:12:37  Writing runtime python for node "model.my_dbt_project.my_python_model"
01:12:37  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=0.6901419162750244s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:12:37  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=0.6904182434082031s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:12:37  On model.my_dbt_project.my_python_model: 
  
    
# models/my_python_model.py
import pandas as pd
def model(dbt, session):
    dbt.config(
        submission_method="all_purpose_cluster",
        cluster_id="0426-154059-dox4kjfg"
    )
    return pd.DataFrame({"id": [1]})


# This part is user provided model code
# you will need to copy the next section to run the code
# COMMAND ----------
# this part is dbt logic for get ref work, do not modify

def ref(*args, **kwargs):
    refs = {}
    key = '.'.join(args)
    version = kwargs.get("v") or kwargs.get("version")
    if version:
        key += f".v{version}"
    dbt_load_df_function = kwargs.get("dbt_load_df_function")
    return dbt_load_df_function(refs[key])


def source(*args, dbt_load_df_function):
    sources = {}
    key = '.'.join(args)
    return dbt_load_df_function(sources[key])


config_dict = {}


class config:
    def __init__(self, *args, **kwargs):
        pass

    @staticmethod
    def get(key, default=None):
        return config_dict.get(key, default)

class this:
    """dbt.this() or dbt.this.identifier"""
    database = "hive_metastore"
    schema = "dbt_jyeo"
    identifier = "my_python_model"
    
    def __repr__(self):
        return '`hive_metastore`.`dbt_jyeo`.`my_python_model`'


class dbtObj:
    def __init__(self, load_df_function) -> None:
        self.source = lambda *args: source(*args, dbt_load_df_function=load_df_function)
        self.ref = lambda *args, **kwargs: ref(*args, **kwargs, dbt_load_df_function=load_df_function)
        self.config = config
        self.this = this()
        self.is_incremental = False

# COMMAND ----------

# how to execute python model in notebook
# dbt = dbtObj(spark.table)
# df = model(dbt, spark)


# --- Autogenerated dbt materialization code. --- #
dbt = dbtObj(spark.table)
df = model(dbt, spark)

import pyspark

# make sure pyspark.sql.connect.dataframe exists before using it
try:
    import pyspark.sql.connect.dataframe
    newer_pyspark_available = True
except ImportError:
    newer_pyspark_available = False

# make sure pandas exists before using it
try:
    import pandas
    pandas_available = True
except ImportError:
    pandas_available = False

# make sure pyspark.pandas exists before using it
try:
    import pyspark.pandas
    pyspark_pandas_api_available = True
except ImportError:
    pyspark_pandas_api_available = False

# make sure databricks.koalas exists before using it
try:
    import databricks.koalas
    koalas_available = True
except ImportError:
    koalas_available = False


# preferentially convert pandas DataFrames to pandas-on-Spark or Koalas DataFrames first
# since they know how to convert pandas DataFrames better than `spark.createDataFrame(df)`
# and converting from pandas-on-Spark to Spark DataFrame has no overhead

if pandas_available and isinstance(df, pandas.core.frame.DataFrame):
    if pyspark_pandas_api_available:
        df = pyspark.pandas.frame.DataFrame(df)
    elif koalas_available:
        df = databricks.koalas.frame.DataFrame(df)

# convert to pyspark.sql.dataframe.DataFrame
if isinstance(df, pyspark.sql.dataframe.DataFrame):
    pass  # since it is already a Spark DataFrame
elif newer_pyspark_available and isinstance(df, pyspark.sql.connect.dataframe.DataFrame):
    pass  # since it is already a Spark DataFrame
elif pyspark_pandas_api_available and isinstance(df, pyspark.pandas.frame.DataFrame):
    df = df.to_spark()
elif koalas_available and isinstance(df, databricks.koalas.frame.DataFrame):
    df = df.to_spark()
elif pandas_available and isinstance(df, pandas.core.frame.DataFrame):
    df = spark.createDataFrame(df)
else:
    msg = f"{type(df)} is not a supported type for dbt Python materialization"
    raise Exception(msg)

writer = (
    df.write
        .mode("overwrite")
        .option("overwriteSchema", "true")
        .format("delta")
)

writer.saveAsTable("`hive_metastore`.`dbt_jyeo`.`my_python_model`")

  
01:12:39  Databricks adapter: Job submission response={'id': '256b07ec2a2643d888cf0412052774a1'}
01:13:00  Databricks adapter: No cursor was provided. Query ID not available.
01:13:00  Execution status: OK in 22.809999465942383 seconds
01:13:00  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=23.52406597137451s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:13:00  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=23.524718046188354s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:13:00  Using databricks connection "model.my_dbt_project.my_python_model"
01:13:00  On model.my_dbt_project.my_python_model: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "node_id": "model.my_dbt_project.my_python_model"} */
describe table `hive_metastore`.`dbt_jyeo`.`my_python_model`
  
01:13:00  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:13:00  SQL status: OK in 0.5699999928474426 seconds
01:13:00  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-de9a-1b80-a9da-1c98e2ddebc9) - Closing cursor
01:13:01  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=24.113465070724487s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:13:01  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=24.113868951797485s, acquire-count=1, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:13:01  Using databricks connection "model.my_dbt_project.my_python_model"
01:13:01  On model.my_dbt_project.my_python_model: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "node_id": "model.my_dbt_project.my_python_model"} */

    
        alter table `hive_metastore`.`dbt_jyeo`.`my_python_model` change column
            id
            comment 'Column description.';
      
  
01:13:01  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:13:03  SQL status: OK in 2.7699999809265137 seconds
01:13:03  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-def4-15ff-942e-b8fd3c24eee6) - Closing cursor
01:13:03  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=26.90058398246765s, acquire-count=0, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Released connection
01:13:03  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=26.90096402168274s, acquire-count=0, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Released connection
01:13:03  Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '88350355-ab6d-4ace-8c7f-b78c15812d11', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x16c1767d0>]}
01:13:03  1 of 2 OK created python table model dbt_jyeo.my_python_model .................. [OK in 26.88s]
01:13:03  Finished running node model.my_dbt_project.my_python_model
01:13:03  Began running node model.my_dbt_project.my_sql_model
01:13:03  2 of 2 START sql table model dbt_jyeo.my_sql_model ............................. [RUN]
01:13:03  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_python_model, idle-time=26.903092861175537s, acquire-count=0, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:13:03  Re-using an available connection from the pool (formerly model.my_dbt_project.my_python_model, now model.my_dbt_project.my_sql_model)
01:13:03  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=26.903486013412476s, acquire-count=0, language=python, thread-identifier=(44871, 6235451392), compute-name=) - Reusing connection previously named model.my_dbt_project.my_python_model
01:13:03  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=26.90369701385498s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Acquired connection on thread (44871, 6235451392), using default compute resource for model '`hive_metastore`.`dbt_jyeo`.`my_sql_model`'
01:13:03  Began compiling node model.my_dbt_project.my_sql_model
01:13:03  Writing injected SQL for node "model.my_dbt_project.my_sql_model"
01:13:03  Began executing node model.my_dbt_project.my_sql_model
01:13:03  MATERIALIZING TABLE
01:13:03  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=26.909281015396118s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:13:03  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=26.909546852111816s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:13:03  Using databricks connection "model.my_dbt_project.my_sql_model"
01:13:03  On model.my_dbt_project.my_sql_model: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "node_id": "model.my_dbt_project.my_sql_model"} */

      describe extended `hive_metastore`.`dbt_jyeo`.`my_sql_model`
  
01:13:03  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:13:04  SQL status: OK in 0.5899999737739563 seconds
01:13:04  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-e09d-10fd-8233-1a2e87da8b8b) - Closing cursor
01:13:04  Writing runtime sql for node "model.my_dbt_project.my_sql_model"
01:13:04  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=27.530657052993774s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:13:04  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=27.531059980392456s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:13:04  Using databricks connection "model.my_dbt_project.my_sql_model"
01:13:04  On model.my_dbt_project.my_sql_model: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "node_id": "model.my_dbt_project.my_sql_model"} */

  
    
        create or replace table `hive_metastore`.`dbt_jyeo`.`my_sql_model`
      
      using delta
      
      
      
      
      
      comment 'Model description.'
      
      
      as
      -- models/my_sql_model.sql
select 1 id
  
01:13:04  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:13:06  SQL status: OK in 2.5 seconds
01:13:06  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-e0fe-123d-b80b-e77b187f430e) - Closing cursor
01:13:06  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=30.03794288635254s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:13:06  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=30.038735151290894s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:13:06  Using databricks connection "model.my_dbt_project.my_sql_model"
01:13:06  On model.my_dbt_project.my_sql_model: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "node_id": "model.my_dbt_project.my_sql_model"} */
describe table `hive_metastore`.`dbt_jyeo`.`my_sql_model`
  
01:13:06  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:13:07  SQL status: OK in 0.550000011920929 seconds
01:13:07  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-e28c-14f4-bd36-ed728ba93b00) - Closing cursor
01:13:07  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=30.59819483757019s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Checking idleness
01:13:07  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=30.5989727973938s, acquire-count=1, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Retrieving connection
01:13:07  Using databricks connection "model.my_dbt_project.my_sql_model"
01:13:07  On model.my_dbt_project.my_sql_model: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "all", "target_name": "db", "node_id": "model.my_dbt_project.my_sql_model"} */

    
        alter table `hive_metastore`.`dbt_jyeo`.`my_sql_model` change column
            id
            comment 'Column description.';
      
  
01:13:07  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=Unknown) - Created cursor
01:13:08  SQL status: OK in 0.8100000023841858 seconds
01:13:08  Databricks adapter: Cursor(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, command-id=01ef4959-e2d1-160f-a6bf-0490e306fb4d) - Closing cursor
01:13:08  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=1.9073486328125e-06s, acquire-count=0, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Released connection
01:13:08  Databricks adapter: DatabricksDBTConnection(id=6107359184, session-id=01ef4959-cf1a-159c-a275-c839a28cd51c, name=model.my_dbt_project.my_sql_model, idle-time=9.5367431640625e-07s, acquire-count=0, language=sql, thread-identifier=(44871, 6235451392), compute-name=) - Released connection
01:13:08  Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '88350355-ab6d-4ace-8c7f-b78c15812d11', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x16a6f1750>]}
01:13:08  2 of 2 OK created sql table model dbt_jyeo.my_sql_model ........................ [OK in 4.52s]
01:13:08  Finished running node model.my_dbt_project.my_sql_model
01:13:08  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=31.39988398551941s, acquire-count=0, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Checking idleness
01:13:08  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=31.400357007980347s, acquire-count=0, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Reusing connection previously named master
01:13:08  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=31.401036024093628s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Acquired connection on thread (44871, 8670427840), using default compute resource
01:13:08  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=31.40168595314026s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Checking idleness
01:13:08  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=None, name=master, idle-time=31.40227699279785s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Retrieving connection
01:13:08  On master: ROLLBACK
01:13:08  Opening a new connection, currently in state init
01:13:09  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=01ef4959-e3b1-1720-89a2-046891a5cc6d, name=master, idle-time=5.0067901611328125e-06s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Connection created
01:13:09  Databricks adapter: NotImplemented: rollback
01:13:09  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=01ef4959-e3b1-1720-89a2-046891a5cc6d, name=master, idle-time=0.0015442371368408203s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Checking idleness
01:13:09  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=01ef4959-e3b1-1720-89a2-046891a5cc6d, name=master, idle-time=0.002103090286254883s, acquire-count=1, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Retrieving connection
01:13:09  Spark adapter: NotImplemented: add_begin_query
01:13:09  Spark adapter: NotImplemented: commit
01:13:09  Databricks adapter: DatabricksDBTConnection(id=6089807376, session-id=01ef4959-e3b1-1720-89a2-046891a5cc6d, name=master, idle-time=2.1457672119140625e-06s, acquire-count=0, language=None, thread-identifier=(44871, 8670427840), compute-name=) - Released connection
01:13:09  Connection 'master' was properly closed.
01:13:09  On master: ROLLBACK
01:13:09  Databricks adapter: NotImplemented: rollback
01:13:09  On master: Close
01:13:09  Databricks adapter: Connection(session-id=01ef4959-e3b1-1720-89a2-046891a5cc6d) - Closing connection
01:13:09  Connection 'model.my_dbt_project.my_sql_model' was properly closed.
01:13:09  On model.my_dbt_project.my_sql_model: ROLLBACK
01:13:09  Databricks adapter: NotImplemented: rollback
01:13:09  On model.my_dbt_project.my_sql_model: Close
01:13:09  Databricks adapter: Connection(session-id=01ef4959-cf1a-159c-a275-c839a28cd51c) - Closing connection
01:13:09  
01:13:09  Finished running 2 table models in 0 hours 0 minutes and 36.85 seconds (36.85s).
01:13:09  Command end result
01:13:09  
01:13:09  Completed successfully

Important bit from the above logs is that:

  • We didn't have any DDL to add table level comments to the python model.
  • We created the SQL table with a table level comment.
  • We had a DDL to add a comment to the python model column.
  • We had a DDL to add a comment to the SQL model column.

Expected behavior

We should add python model (table) level descriptions just like we do for SQL models.

Screenshots and log output

image image

System information

The output of dbt --version:

Core:
  - installed: 1.8.4
  - latest:    1.8.4 - Up to date!

Plugins:
  - databricks: 1.8.4 - Up to date!

The operating system you're using:
macOS

The output of python --version:
3.11.9

Additional context
@jeremyyeo jeremyyeo added the bug Something isn't working label Jul 24, 2024
@benc-db
Copy link
Collaborator

benc-db commented Jul 24, 2024

Thanks for reporting

@benc-db benc-db mentioned this issue Jul 24, 2024
Merged
3 tasks
elca-anh added a commit to elca-anh/dbt-databricks that referenced this issue Aug 20, 2024
@elca-anh
update change log with new feature databricks#740
a0dd8c5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

When we persist doc, for python lets remember to add a comment databricks/dbt-databricks
2 participants
@jeremyyeo @benc-db