Add DatabricksSQLStatementsOperator #48507
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pankajkoti
force-pushed
the
add-databricks-sql-statements-operator-with-async
branch
2 times, most recently
from
1a49b2c to
2b079d3
Compare
2 tasks
pankajkoti
force-pushed
the
add-databricks-sql-statements-operator-with-async
branch
from
2b079d3 to
d5b07d8
Compare
Member
Author
|
cc: @pankajastro @phanikumv @tatiana tagging for seeking review on the PR |
rawwar
reviewed
Apr 2, 2025
providers/databricks/src/airflow/providers/databricks/hooks/databricks.py
Outdated
Show resolved
Hide resolved
rawwar
reviewed
Apr 2, 2025
providers/databricks/src/airflow/providers/databricks/hooks/databricks.py
Outdated
Show resolved
Hide resolved
phanikumv
reviewed
Apr 2, 2025
pankajastro
approved these changes
Apr 2, 2025
providers/databricks/src/airflow/providers/databricks/hooks/databricks.py
Show resolved
Hide resolved
providers/databricks/src/airflow/providers/databricks/triggers/databricks.py
Show resolved
Hide resolved
providers/databricks/src/airflow/providers/databricks/operators/databricks.py
Show resolved
Hide resolved
providers/databricks/src/airflow/providers/databricks/operators/databricks.py
Show resolved
Hide resolved
providers/databricks/src/airflow/providers/databricks/operators/databricks.py
Show resolved
Hide resolved
pankajkoti
force-pushed
the
add-databricks-sql-statements-operator-with-async
branch
2 times, most recently
from
260a0be to
42ab985
Compare
pankajkoti
force-pushed
the
add-databricks-sql-statements-operator-with-async
branch
from
6e7447f to
46d01f3
Compare
rawwar
approved these changes
Apr 3, 2025
2 tasks
nailo2c
pushed a commit
to nailo2c/airflow
that referenced
this pull request
Apr 4, 2025
This PR introduces the DatabricksSQLStatementsOperator to the Databricks provider, enabling the submission of dynamic SQL statements and polling their execution status. Additionally, it includes support for deferrable mode. The operator utilises the [Databricks Statement Execution API](https://docs.databricks.com/api/workspace/statementexecution), allowing SQL submission via the [POST](https://docs.databricks.com/api/workspace/statementexecution/executestatement) endpoint and subsequent polling using the `statement ID` (returned by the POST endpoint) through the [GET](https://docs.databricks.com/api/workspace/statementexecution/getstatement) endpoint. closes: apache#46549
diogotrodrigues
pushed a commit
to diogotrodrigues/airflow
that referenced
this pull request
Apr 6, 2025
This PR introduces the DatabricksSQLStatementsOperator to the Databricks provider, enabling the submission of dynamic SQL statements and polling their execution status. Additionally, it includes support for deferrable mode. The operator utilises the [Databricks Statement Execution API](https://docs.databricks.com/api/workspace/statementexecution), allowing SQL submission via the [POST](https://docs.databricks.com/api/workspace/statementexecution/executestatement) endpoint and subsequent polling using the `statement ID` (returned by the POST endpoint) through the [GET](https://docs.databricks.com/api/workspace/statementexecution/getstatement) endpoint. closes: apache#46549
simonprydden
pushed a commit
to simonprydden/airflow
that referenced
this pull request
Apr 8, 2025
This PR introduces the DatabricksSQLStatementsOperator to the Databricks provider, enabling the submission of dynamic SQL statements and polling their execution status. Additionally, it includes support for deferrable mode. The operator utilises the [Databricks Statement Execution API](https://docs.databricks.com/api/workspace/statementexecution), allowing SQL submission via the [POST](https://docs.databricks.com/api/workspace/statementexecution/executestatement) endpoint and subsequent polling using the `statement ID` (returned by the POST endpoint) through the [GET](https://docs.databricks.com/api/workspace/statementexecution/getstatement) endpoint. closes: apache#46549
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the DatabricksSQLStatementsOperator to the Databricks provider, enabling the submission of dynamic SQL statements and polling their execution status. Additionally, it includes support for deferrable mode.
The operator utilises the Databricks Statement Execution API, allowing SQL submission via the POST endpoint and subsequent polling using the
statement ID(returned by the POST endpoint) through the GET endpoint.closes: #46549