Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Databricks operators: added support for pipeline task, jar_params, example DAG for RunNowOperator, and updated docs #19156

Closed
wants to merge 1 commit into from

Conversation

robertsaxby
Copy link
Contributor

Added support for pipeline task, corrected documentation urls

@abij
Copy link
Contributor

abij commented Oct 25, 2021

Thx, I was confused that it's not the latest version (2.1), but v 2.0 it's missing the access_controll_list property for job submission for example.

This makes the docs inline with code.

@robertsaxby
Copy link
Contributor Author

Thx, I was confused that it's not the latest version (2.1), but v 2.0 it's missing the access_controll_list property for job submission for example.

This makes the docs inline with code.

yes this is still v2.0 (as before, even though some of the doc links incorrectly pointed to latest-which is v2.1) of the Databricks Jobs API. I will submit another update later today to include the jar_params in the RunNow operators plus some examples and doc changes.

The work to upgrade to v2.1 of the Databricks Jobs API is still oustanding.

@robertsaxby robertsaxby changed the title Updated Databricks operator Updated Databricks operator, example dags and docs Oct 25, 2021
@lennartkats-db
Copy link

Good point about the 2.0 API, @abij. The goal here is to just improve the current operator documentation and fill some minor gaps. The same feature set would be supported on 2.1 but that migration can be covered separately.

@lennartkats-db
Copy link

@robertsaxby Static checks appear to be failing, please review

Copy link

@lennartkats-db lennartkats-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from Databricks perspective, minor question around version numbers. Also, please note that one test still appears to fail, please review.

airflow/providers/databricks/provider.yaml Outdated Show resolved Hide resolved
@robertsaxby robertsaxby changed the title Updated Databricks operator, example dags and docs Databricks operators: added support for pipeline task, jar_params, example DAG for RunNowOperator, and updated docs Oct 26, 2021
@alexott
Copy link
Contributor

alexott commented Dec 5, 2021

pipeline_task and jar_params where added in #19443.

@potiuk
Copy link
Member

potiuk commented Dec 5, 2021

Needs rebase!

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jan 20, 2022
@github-actions github-actions bot closed this Jan 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers stale Stale PRs per the .github/workflows/stale.yml policy file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants