Update Databricks API from 2.0 to 2.1

[kyle-hamlin]

This PR updates the Databricks API version from 2.0 to 2.1. Of the nine available endpoints in the DatabricksHook, these seven are not in use and do not map to any operators.

[NOT IN USE] api/2.1/clusters/restart => None
[NOT IN USE] api/2.1/clusters/start => None
[NOT IN USE] api/2.1/clusters/delete => None
[NOT IN USE] api/2.1/jobs/runs/get => None
[NOT IN USE] api/2.1/jobs/runs/cancel => None
[NOT IN USE] api/2.1/libraries/install => None
[NOT IN USE] api/2.1/libraries/uninstall => None

The remaining two endpoints map to a single operator each and the request structure is either unchanged or compatible when moving from version 2.0 to 2.1 according to this doc https://docs.databricks.com/data-engineering/jobs/jobs-api-updates.html#api-client-guide

[COMPATIABLE] api/2.1/jobs/runs/submit => DatabricksSubmitRunOperator
[UNCHANGED] api/2.1/jobs/run-now  => DatabricksRunNowOperator

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

Yeah @Mokubyow - can you please remove the changelog entry ?

Also - question (just to confirm) - the 2.1 API is fully backwards compatible with 2.0 in the way we use it ?

[alexott]

It would be nice to add support for multi-task jobs which is primary reason for 2.1 API... Although maybe it makes sense to wait until for a new functionality

[potiuk]

It would be nice to add support for multi-task jobs which is primary reason for 2.1 API... Although maybe it makes sense to wait until for a new functionality

Yeah. Let's split it to separate PR.

[alexott]

I agree. And it will makes sense to wait a bit - there will be improvements in the near future

[kyle-hamlin]

@potiuk Yes, the two endpoints we use are fully backward compatible/unchanged as noted in the documentation I linked.

@alexott Happy to help with adding that support on another PR :)

[potiuk]

Some statuc checks failing @Mokubyow - I recommend using pre-commit to fix them

[kyle-hamlin]

@potiuk Looks like the pre-commit worked

[boring-cyborg]

Awesome work, congrats on your first merged pull request!

[christophergrant]

Any APIs that are not under the api/2.x/jobs prefix do not have version 2.1 at this time i.e the clusters APIs and libraries APIs will need to be reverted to 2.0.