[FEATURE] Support creating notification destinations

[Esya]

Use-cases

Right now it is possible when creating a Job to set webhook_notifications to an array of notification destination IDs, such as :

"on_failure": [ { "id": "0481e838-0a59-4eff-9541-a4ca6f149574" } ]

But there is no way to create those notification destinations through the provider. We create whole workspaces using Terraform and configure them entirely - and we would like to have Slack alerts for jobs set-up automatically on every job.

Attempted Solutions

Right now the only solution is to manually create the destinations in the workspaces, save the IDs, and use those IDs in Terraform.
The problem with this is that we cannot create an entirely new workspace in one run.

Proposal

Have an alert_destination resource that matches what's visible in the UI, for example for Slack, Name and Webhook URL

It seems that the API used by the Workspace UI to create alert destinations is not documented though, and it does not appear in the REST API (Latest) documentation unless I'm missing something ?

[alexott]

it's not implemented because there is no public API for that functionality. As soon as it will be added, new resource will be created

[fdvieira]

Hello,

Any news on this? We would like to use this in terraform resource!
Francisco

[alexott]

No public API still, and no ETA yet

[rondefreitas]

It looks like the feature itself is now GA. Any word on the API required to leverage it through GO and terraform?

[alexott]

Nope

[sbchapin]

Anyone who comes here from the public and looks at this issue, there are two things you can do:

1. Ask your Databricks rep to add management of notification destinations to the Databricks public API.
2. Vote for this yourself via ideas.databricks.com - an idea for it has already been raised, so go vote for it!

[jonathan-vogel-siemens]

@sbchapin any way to access ideas.databricks.com when using Azure Databricks?

[nchammas]

If you have some existing notification destinations, which I am currently looking at via this URL,

https://<workspace-name>.cloud.databricks.com/settings/workspace/notifications/notification-destinations

is there some way to get the id of one of those existing destinations?

The documentation for webhook_notifications says:

The id can be retrieved through the API or the URL of Databricks UI https://<workspace host>/sql/destinations/<notification id>?o=<workspace id>

But this doesn't make sense since this URL contains the notification ID we're trying to find in the first place! And I can't see a way via the Databricks CLI or REST API to find the ID of an existing notification destination.

Is it even possible?

I agree with the original issue that it would best if we could manage notification destinations using Terraform. However, I'm struggling to even use existing notification destinations via Terraform.

[nchammas]

OK, a colleague pointed out that this information is available in the web UI behind this nondescript "copy" icon:

Still can't find a way to get this same information via the CLI or REST API (as expected), but at least we can do it manually.

[alexott]

API is now available: https://docs.databricks.com/api/workspace/notificationdestinations, but we need to get new Go SDK released to work on it.

[alexott]

Closing this as #3820 is merged now