Upgrade the Airflow Databricks provider to use the Databricks jobs API 2.2

[JonasDoubleU]

Description

Upgrade the Airflow Databricks provider to use the Databricks jobs API 2.2 from 2.1. Specifically the GET_RUN_ENDPOINT, to allow fetching of the status of all tasks within a workflow with more than 100 tasks. Consequently, the fetching probably needs to be modifed to handle pagination.

Use case/motivation

We are using Airflow to trigger Databricks workflows, but some of our workflows contain more than 100 tasks. The support to create workflows with more than 100 tasks with the Databricks jobs API was added in version 2.2. Therefore, when fetching the status of the tasks Airflow fails because it uses version 2.1. Ultimately, we cannot use Airflow to trigger these workflows with the current Airflow Databricks provider.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jroachgolf84]

Happy to take a look at this one!

[jroachgolf84]

I've created a draft PR to upgrade to API v2.2 for all jobs endpoints. All unit-tests are passing (after updating to point to the proper version). However, in order for this to work, we'd need to update the operator to handle more than 100 tasks.

I have a few questions for @jscheffl and @eladkal:

• Are there other tests that need to be done for changes like this?
• Should we update other endpoints point to v2.0 and v2.1?

[jscheffl]

I have a few questions for @jscheffl and @eladkal:

* Are there other tests that need to be done for changes like this?

* Should we update other endpoints point to v2.0 and v2.1?

Unfortunately I am not an expert in databricks API so I have no proper answer for you.

I am not sure what effort would be needed to support more than 100 tasks but also I'd think it might be OK to limit it to 100 for the moment and mark a todo/add a follow-up issue to split the work to a separate PR and not this is maybe a current limitation. Not sure how many DBX workflows really exceed this number. Upgrading for API compatibility sounds more important to me than having "no limits".

Regarding the other endpoints, I'd also assume this would be okay to have a s a follow-up

Also in general I'd hope that the team of DBX itself would have a high interest to keep the integration up-to-date. So alongside the providers contribution model this is a point where I'd expect some DBX maintainers to jump-in.

[JonasDoubleU]

The handling of more than 100 tasks is the crucial part of this issue for us to be able to use Airflow. I am happy to open another issue for that functionality if it would be better.

[jroachgolf84]

@JonasDoubleU - I've completed the PR that will upgrade to v2.2. Once that's approved, I'll go ahead and create the PR to handle more than 100 tasks.

[dzmitry-belavus]

Hello!
We've tried the new release version (7.8.2) of the provider and it gives 404 for some of the 2.2 endpoints:

Response: {"error":"Bad Target: /api/2.2/permissions/jobs/{job_id}"}, Status Code: 404

[aru-trackunit]

#60444