Input from dbt Labs's core adapters team

[dataders]

as part of our new adapter verification process, the dbt core adapters engineering team and I met today to do a high-level review of the dbt-databricks codebase.

good news, your adapter passes the sniff test! That means that while the engineers haven’t yet had the time to review every single line of code (though they plan to in the next few sprints), we did not see anything that should stop the verification process.

An interesting output is that the team found opportunities where they might be better able to support, both of which are not of an urgent priority to fix:

1. the code introduced in Block taking jinja2.runtime.Undefined into DatabricksAdapter #98 was difficult to grok at first (TIL . is this to support both UC (that have database/catalog) and no-UC dbt-databricks (that do not have database/catalog) projects? I imagine that conditionality can pretty tricky, and the core team wanted to meet with y’all to discuss potential simpler solutions
2. this parsing of our exception message (see below) raised our hackles, in that now we know that things will break if we decide to standardize our Exception handling messages as we have already done with our logging. That said, we recognize the need for it, but wanted to tell you that we do plan to overhaul our Exception handling so that you won't even need this logic moving forward. stay tuned.
   

   dbt-databricks/dbt/adapters/databricks/impl.py

   Lines 114 to 125 in dbd58fb

   	try:
   	# The catalog for `show table extended` needs to match the current catalog.
   	with self._catalog(schema_relation.database):
   	results = self.execute_macro(LIST_RELATIONS_MACRO_NAME, kwargs=kwargs)
   	except dbt.exceptions.RuntimeException as e:
   	errmsg = getattr(e, "msg", "")
   	if f"Database '{schema_relation}' not found" in errmsg:
   	return []
   	else:
   	description = "Error while retrieving information about"
   	logger.debug(f"{description} {schema_relation}: {e.msg}")
   	return []

lastly, are there any particular areas you’d like us to review more closely? Are there any ways that the dbt-core adapters team can better support dbt-databricks now and in the future? let us know!

[gatorsmile]

@dataders Thank you!

What is the new adapter verification process? Could you provide more info about the process?

[dataders]

So here's what we have public-facing right now Verifying a new adapter. Reach out to @superdupershant and he can give you more info that we've shared with him.

[dataders]

We also noticed that the integration test suite from dbt-core is defined, but not being run as part of CI (see #204). If there was one thing to fix before the dbt-databricks is officially verified, it is this. 👀 @gatorsmile @ueshin @superdupershant

[ueshin]

1. the code introduced in Block taking jinja2.runtime.Undefined into DatabricksAdapter #98 was difficult to grok at first (TIL . is this to support both UC (that have database/catalog) and no-UC dbt-databricks (that do not have database/catalog) projects? I imagine that conditionality can pretty tricky, and the core team wanted to meet with y’all to discuss potential simpler solutions

Yes, jinja2.runtime.Undefined was troublesome when supporting both UC and non-UC. For example, we needed to make the include policy for database be True, but if it's Undefined, the rendered string will be .schema.table and cause to fail the query. #98 tries to replace Undefined with None when the macro calls @available adapter methods.

It would be great if dbt-core could handle or provide the way to handle Undefined whenever macros call Python functions/methods.