feat: add Databricks Serverless support #3001
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eakmanrq
force-pushed
the
eakmanrq/add_databricks_serverless_support
branch
2 times, most recently
from
d1679a1
to
da254fe
Compare
izeigerman
reviewed
Aug 12, 2024
izeigerman
approved these changes
Aug 12, 2024
|
Found out that when using Databricks SQL Connector + Databricks connect then you need the global temp views since they are trying to share temp objects across sessions (but still within the same application). Therefore now I just use the global temp views in all cases except when using databricks serverless. Also added serverless support to databricks-connect. |
izeigerman
reviewed
Aug 13, 2024
sqlmesh/core/engine_adapter/spark.py
Outdated
| @@ -260,8 +264,14 @@ def _df_to_source_queries( | |||
|
|
|||
| def query_factory() -> Query: | |||
| temp_table = self._get_temp_table(target_table or "spark", table_only=True) | |||
| df.createOrReplaceGlobalTempView(temp_table.sql(dialect=self.dialect)) # type: ignore | |||
| temp_table.set("db", "global_temp") | |||
| if self.use_serverless: | |||
There was a problem hiding this comment.
This looks like databricks specific. Should we just override query_factory it in the corresponding adapter?
There was a problem hiding this comment.
EG.:
def query_factory() -> Query:
temp_table = self._get_temp_table(target_table or "spark", table_only=True)
if self.use_serverless:
...
else:
super()....
There was a problem hiding this comment.
Hmm I think I will just completely override it since the _use_spark_session is also databricks specific.
Change: 91cb2a1
eakmanrq
force-pushed
the
eakmanrq/add_databricks_serverless_support
branch
3 times, most recently
from
3cd5b9b
to
52e8a8e
Compare
eakmanrq
force-pushed
the
eakmanrq/add_databricks_serverless_support
branch
from
52e8a8e
to
4417423
Compare
izeigerman
approved these changes
Aug 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds support for using Serverless compute from a notebook. I tested by plan/applying using a Serverless environment.
Two key changes:
Note: This PR is build on the current constraints that the Python SQL Connector cannot query serverless. If that changes in the future, then the fix in the PR of having serverless use session temp views would become a problem if you want to mix SQL Connector and Databricks Connect. It might be that the global temp constraint is lifted by then but if not then the hybrid mode of using both would likely need to be disabled when using serverless.