Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Databricks] Fix issue for using single notebook task launch without taskgroup #162

Merged
merged 8 commits into from
Feb 28, 2023

Conversation

pankajkoti
Copy link
Contributor

The latest Databricks Jobs API 2.1 offers a consistent response for
both single task and multi-task job run submit API which fixes the current
issue mentioned in the ticket.

The PR also adds an example DAG for running the DatabricksNotebookOperator
alone without the DatabricksWorkflowTaskGroup operator.

closes: #160

@pankajkoti pankajkoti requested a review from a team as a code owner February 25, 2023 16:49
@pankajkoti pankajkoti self-assigned this Feb 25, 2023
Copy link
Contributor

@dimberman dimberman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like @tatiana said, once there are unit tests I'm glad to approve.

cosmos/providers/databricks/notebook.py Show resolved Hide resolved
@pankajkoti pankajkoti force-pushed the 160-run-single-notebook-task branch from d3c50ed to 6f2d19b Compare February 28, 2023 09:30
@pankajkoti
Copy link
Contributor Author

Like @tatiana said, once there are unit tests I'm glad to approve.

@dimberman I have added them now, please check.

@pankajkoti pankajkoti dismissed dimberman’s stale review February 28, 2023 11:17

Feedback taken and addressed.

@pankajkoti pankajkoti requested a review from tatiana February 28, 2023 12:17
@pankajkoti pankajkoti force-pushed the 160-run-single-notebook-task branch from 2da9b5f to 5c13751 Compare February 28, 2023 14:12
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch and quick fix, @pankajkoti , thank you!

@tatiana tatiana merged commit 8f32304 into main Feb 28, 2023
@tatiana tatiana deleted the 160-run-single-notebook-task branch February 28, 2023 14:14
@pankajkoti pankajkoti modified the milestone: Databricks Workflow support Feb 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Databricks] Running a Notebook task without WorkflowTaskGroup fails
3 participants