issue-59189: Updating Databricks provider to point to 2.2/jobs/... endpoint

[jroachgolf84]

Description

The apache-airflow-providers-databricks provider uses API version 2.1 for all jobs operations. This PR bumps that API version to 2.2 for all jobs operations. These changes were made in three places:

• Hooks/Operators
• tests...
• Documentation

related: #59189

[jroachgolf84]

Including comment from issue.

I've created a draft PR to upgrade to API v2.2 for all jobs endpoints. All unit-tests are passing (after updating to point to the proper version). However, in order for this to work, we'd need to update the operator to handle more than 100 tasks.

I have a few questions for @jscheffl and @eladkal:

• Are there other tests that need to be done for changes like this?
• Should we update other endpoints point to v2.0 and v2.1?

[jscheffl]

Including comment from issue.

I've created a draft PR to upgrade to API v2.2 for all jobs endpoints. All unit-tests are passing (after updating to point to the proper version). However, in order for this to work, we'd need to update the operator to handle more than 100 tasks.

I have a few questions for @jscheffl and @eladkal:

* Are there other tests that need to be done for changes like this?

* Should we update other endpoints point to v2.0 and v2.1?

Now that I see the PR (besides the comments I made on the issue) I think it is straight forward to upgrade all endpoints to 2.2 consistently.

[potiuk]

Now that I see the PR (besides the comments I made on the issue) I think it is straight forward to upgrade all endpoints to 2.2 consistently.

Yep.

[jroachgolf84]

Sounds good, I'll go ahead and update all of those endpoints.

[jroachgolf84]

@eladkal, do you know the folks who maintain the Databricks provider? I'd love to get their review on this.

[potiuk]

@eladkal, do you know the folks who maintain the Databricks provider? I'd love to get their review on this.

There is - currently - no dedicated people to review providers - we we want to introduce it as a way to track it - see https://lists.apache.org/thread/jqjd801g9161dqyv7l2xbsw0v7mp77z1 but it has not happened yet.

Your best guess is to look at git history for the provider

[jroachgolf84]

Thanks, @potiuk.

[jscheffl]

Let's do a final rebase and then LGTM in my view.

[aru-trackunit]

This endpoint unfortunately does not exist and I even do think that has been migrated to: https://docs.databricks.com/api/workspace/warehouses/list

[2026-01-13 03:32:41] ERROR - Task failed with exception source=task loc=task_runner.py:986
AirflowException: Response: {"error":"Bad Target: /api/2.2/sql/endpoints"}
, Status Code: 404

HTTPError: 404 Client Error: Not Found for url: https://dbc-instance-random.cloud.databricks.com/api/2.2/sql/endpoints

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 704 in _do_api_call
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 445 in __iter__
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378 in iter
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400 in <lambda>
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449 in result
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401 in __get_result
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 723 in _do_api_call
File "/home/airflow/.local/lib/python3.12/site-packages/requests/models.py", line 1026 in raise_for_status

Previous version api/2.0/sql/endpoints works is there even though is not listed in the Databricks documentation.

[aru-trackunit]

Hi! Apart of updating jobs api path to /api/2.2 you also updated /api/2.2/sql/endpoints that does not exist, so I am getting:

[2026-01-13 03:32:41] ERROR - Task failed with exception source=task loc=task_runner.py:986
AirflowException: Response: {"error":"Bad Target: /api/2.2/sql/endpoints"}
, Status Code: 404

HTTPError: 404 Client Error: Not Found for url: https://dbc-instance-random.cloud.databricks.com/api/2.2/sql/endpoints

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 704 in _do_api_call
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 445 in __iter__
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378 in iter
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400 in <lambda>
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449 in result
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401 in __get_result
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 723 in _do_api_call
File "/home/airflow/.local/lib/python3.12/site-packages/requests/models.py", line 1026 in raise_for_status

Previous version api/2.0/sql/endpoints works is there even though is not listed in the Databricks documentation. Just retested it.

Latest Databricks docs does not even include sql/endpoints my guess it has been migrated to sql/warehouses https://docs.databricks.com/api/workspace/warehouses/list

Maybe that’s just me but when doing API version bumps I would expect to bump a minor version not a fix version number.

Should latest databricks provider version be yanked?

[eladkal]

@aru-trackunit can you raise a quick fix?

Should latest databricks provider version be yanked?

If the fix partially works I am not sure we can yank it as there are users who might already using the rest of the functionality. We normally yank if the release is completely invalid.

[aru-trackunit]

Created this: #60444, I see that there are others raising missing endpoints comments, #59189 (comment)

[eladkal]

@aru-trackunit if you know what the problem and what is the needed fix you can just go a head and create a PR with the fix. There is a planned release for later today so if the fix is merge before it will be included

[jscheffl]

I am cutting providers wave already, so too late now. As the bug is already released I assume a fix would be great for the next providers wave in 2 weeks. Except it we judge urgent and we cut a special RC2 for this.