provider Databricks : add cluster API

[ThAccart]

Description

Add Databricks Cluster API in Databricks Hook

Use case/motivation

Databricks provider lacks Cluster API access to allow
to CRUD cluster as well as manage clusters (start/restart/delete/...)
Those API are very convenient to control clusters (get status, force shutdown...) and to optimize some specific flows (scale cluster in advance when a coming need is detected )

Related issues

none

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Assigned you the issue!

[alexott]

It's interesting use case. But really we have two things here:

• cluster creation/deletion - these tasks are nicely done with Terraform, for example
• managing the cluster state - stop/start - that's not possible (or very limited) with Terraform

[alexott]

@potiuk it looks like it's stale, can you assign it to me?

[potiuk]

Sure!

[eladkal]

@alexott are you still working on this issue?

[alexott]

Yes, I do

[Seokyun-Ha]

Hello, guys!! I opened new PR of Databricks ClusterState utility :)

Any feedback would be appreciated!

#34071

[Lee-W]

It seems this issue has not been updated for more than one year and @Seokyun-Ha is not working on it. I assigned this one to @Seokyun-Ha