Add DatabricksWorkflowPlugin

[pankajkoti]

The DatabricksWorkflowPlugin provides with links in the Airflow
UI for tasks that allow us to see the Databricks job run in the
Databricks workspace, additionally it also provides link to
repair task(s) in the workflow.

Databricks does not allow repairing jobs with single tasks launched
outside the workflow, hence we just provide the link for the job run.

Within the workflow, for each of the task, we provide links to the
job run and repair link for the single task

And at the workflow level, for the job launch task, we provide a
link to repair all failed tasks along with the link for job run in
the Databricks workspace that can be used to monitor the job
in the Databricks account.

This PR is the second attempt on adding the DatabricksWorkflowPlugin,
the previous attempt being #40153. However, there were some concerns
raised in #40708 and hence it was reverted in #40714. This newer PR
attempts to address those concerns.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[pankajkoti]

@vincbeck would you have an idea on how I could resolve the CI test failures here.

They complain with

FAILED tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py::TestAwsAuthManager::test_aws_auth_manager_index - werkzeug.routing.exceptions.BuildError: Could not build url for endpoint 'RepairDatabricksTasks.repair'. Did you forget to specify values ['dag_id', 'run_id']?

[vincbeck]

From the logs, all I can see is when Airflow tries to load the index page (DAGs list), it is failing because of:

werkzeug.routing.exceptions.BuildError: Could not build url for endpoint 'RepairDatabricksTasks.repair'. Did you forget to specify values ['dag_id', 'run_id']?

From the stacktrace I can see that it seems your plugin add a new element in the nav bar pointing to the view RepairDatabricksTasks.repair. This view has some mandatory parameter (dag_id and run_id) and that's why it is failing. When trying to build the nav bar and add a link towards this view, Airflow cannot because it requires some parameters.

I am surprised it works when you run Airflow ... Need further investigation

[jscheffl]

Just reviewed the plugin, rest I asusm eis unchanged from previous merged PR.

Some questions on the plugin, I think a lot of validation coul dbe removed with a URL pattern extension.

Otherwise... now looking secure and menu entry is gone. Thanks.

[jscheffl]

Thanks for the efforts in making the plugin more secure.
The errors in build seem to be unrelated - but would be good to double check before merge.

[pankajkoti]

Thanks for the efforts in making the plugin more secure. The errors in build seem to be unrelated - but would be good to double check before merge.

Thanks a lot to you for raising the concerns in the first place, we're now in much better shape :) Yes, the failures are unrelated, I will go ahead with the merge

Thanks to @vincbeck for checking the CI failures during the journey on the PR :)