Add account-level API support for Unity Catalog objects (#2182)

* first draft * account client check * account client check * Fixed `databricks_service_principals` data source issue with empty filter (#2185) * fix `databricks_service_principals` data source issue with empty filter * fix acc tests * Allow rotating `token` block in `databricks_mws_workspaces` resource by only changing `comment` field (#2114) Tested manually for the following cases Without this PR the provider recreates the entire workspace on a token update With changes in this PR only the token is refreshed When both token and storage_configuration_id are changed then the entire workspace is recreated Additional unit tests also added that allow checks that patch workspace calls are not made when only token is changed Also added an integration test to check tokens are successfully updated * Excludes roles in scim API list calls to reduce load on databricks scim service (#2181) * Exclude roles in scim API list calls * more test fixes * Update SDK to v0.6.0 (#2186) * Update SDK to v0.6.0 * go mod tidy * update sdk to 0.7.0 * add integration tests * fix acceptance tests * fix tests * add account-level API support for `metastore_data_access` * add account API support for `databricks_storage_credential` * address feedback * refactor to `WorkspaceOrAccountRequest` * fix acceptance tests * Release v1.21.0 (#2471) Release v1.21.0 of the Terraform Provider for Databricks. ## Changes * Added condition_task to the [`databricks_job`](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource (private preview) ([#2459](#2459)). * Added `AccountData`, `AccountClient` and define generic databricks data utilites for defining workspace and account-level data sources ([#2429](#2429)). * Added documentation link to existing Databricks Terraform modules ([#2439](#2439)). * Added experimental compute field to [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource ([#2401](#2401)). * Added import example to doc for [databricks_group_member](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group_member) resource ([#2453](#2453)). * Added support for subscriptions in dashboards & alert SQL tasks in [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) ([#2447](#2447)). * Fixed model serving integration test ([#2460](#2460), [#2461](#2461)). * Fixed [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource file arrival trigger parameter name ([#2438](#2438)). * Fixed catalog_workspace_binding_test ([#2463](#2463), [#2451](#2451)). No breaking changes in this release. * Install mlflow cluster using in model serving test if the cluster is already running (#2470) * Bump golang.org/x/mod from 0.11.0 to 0.12.0 (#2462) Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.11.0 to 0.12.0. - [Commits](golang/mod@v0.11.0...v0.12.0) --- updated-dependencies: - dependency-name: golang.org/x/mod dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Exporter: make resource names more unique to avoid duplicate resources errors (#2452) This includes following changes: * Add user ID to the `databricks_user` resource name to avoid clashes on names like, `user+1` and `user_1` * Add user/sp/group ID to the name of the `databricks_group_member` resource * Remove too aggressive name normalization pattern that also leads to the generation of the duplicate resource names for different resources * Add documentation notes about legacy cluster type & data access (#2437) * Add documentation notes about legacy cluster type & data access * Update docs/resources/cluster.md Co-authored-by: Miles Yucht <miles@databricks.com> * Update docs/resources/mount.md Co-authored-by: Miles Yucht <miles@databricks.com> --------- Co-authored-by: Miles Yucht <miles@databricks.com> * Use random catalog name in SQL table integration tests (#2473) The fixed value prevented concurrent integration test runs. * Link model serving docs to top level README (#2474) * Add one more item to the troubleshooting guide (#2477) It's related to use OAuth for authentication but not providing `account_id` in the provider configuration. * Added `databricks_access_control_rule_set` resource for managing account-level access (#2371) * Added `acl_principal_id` attribute to `databricks_user`, `databricks_group` & `databricks_service_principal` for easier use with `databricks_access_control_rule_set` (#2485) It should simplify specification of principals in the `databricks_access_control_rule_set` so instead of this (string with placeholders): ``` grant_rules { principals = ["groups/${databricks_group.ds.display_name}"] role = "roles/servicePrincipal.user" } ``` it will be simpler to refer like this: ``` grant_rules { principals = [databricks_group.ds.acl_principal_id] role = "roles/servicePrincipal.user" } ``` * Added support for Unity Catalog `databricks_metastores` data source (#2017) * add documentation for databricks_metastores data source * add API endpoint for listing metastores * add metastores data resource * add test for metastores data source * add metastores datasource to resource mapping * fix reference to wrong resource docs * add a Metastores struct for the response of the API, use this in the datSource * update terraform specific object attributes * add new data test * remove slice_set property from MetastoreData * use databricks-go-sdk for data_metastore.go * removed listMetastores endpoint since it's unused * make sure tests also use the unitycatalog.MetastoreInfo from the sdk * remove redundant resource * test -dev * fix * fmt * cleanup * Added AccountClient to DatabricksClient and AccountData * upd * cleanup * accountLevel * upd * add example * list * cleanup * docs * remove dead code * wip * use maps * upd * cleanup * comments * - * remove redundant test --------- Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com> Co-authored-by: vuong-nguyen <44292934+nkvuong@users.noreply.github.com> * Added support for Unity Catalog `databricks_metastore` data source (#2492) Enable fetching account level metastore information through id for a single metastore. * Supported new Delve binary name format (#2497) https://github.com/go-delve/delve/blob/master/CHANGELOG.md#1210-2023-06-23 changes the naming of the delve debug binary. This PR changes isInDebug to accommodate old and new versions of Delve. * Add code owners for Terraform (#2498) * Removed unused dlvLoadConfig configuration from settings.json (#2499) * Fix provider after updating SDK to 0.13 (#2494) * Fix provider after updating SDK to 0.13 * add unit test * split test * Added `control_run_state` flag to the `databricks_job` resource for continuous jobs (#2466) This PR introduces a new flag, control_run_state, to replace the always_running flag. This flag only applies to continuous jobs. Its behavior is described below: For jobs with pause_status = PAUSED, it is a no-op on create and stops the active job run on update (if applicable). For jobs with pause_status = UNPAUSED, it starts a job run on create and stops the active job run on update (if applicable). The job does not need to be started, as that is handled by the Jobs service itself. This fixes #2130. * Added exporter for `databricks_workspace_file` resource (#2493) * Preliminary changes to make workspace files implementation - make `NotebooksAPI.List` to return directories as well when called in the recursive mode (same as non-recursive behavior) - Because of that, remove the separate `ListDirectories` - Extend `workspace.ObjectStatus` with additional fields (will be required for incremental notebooks export) - Cache listing of all workspace objects, and then use it for all operations - list notebooks, list directories, list workspace files * Added exporting of workspace files --------- Co-authored-by: Miles Yucht <miles@databricks.com> * Supported boolean values in `databricks_sql_alert` alerts (#2506) * Added more common issues for troubleshooting (#2486) * add troubleshooting * fix doc category --------- Co-authored-by: Miles Yucht <miles@databricks.com> * Fixed handling of comments in `databricks_sql_table` resource (#2472) * column comments and single quote escape * Delimiter collision avoidance table comment * compatible with user single quote escape * unit tests for parseComment * corrected fmt --------- Co-authored-by: Miles Yucht <miles@databricks.com> * Added clarification that `databricks_schema` and `databricks_sql_table` should be imported by their full name, not just by name (#2491) Co-authored-by: Miles Yucht <miles@databricks.com> * Updated `databricks_user` with `force = true` to check for error message prefix (#2510) This fixes #2500 * fix force delete * remove orphaned code * fix acceptance tests * upgrade go sdk * fix metastoreinfo struct * docs update * fix acceptance tests * fix tests * updated docs * fix tests * rename test * update tests * fix tests * fix test * add state upgrader * fix struct * fix tests * feedback * feedback * fix acc test * fix test * fix test * fix test * feedback * fix acc tests * feedback --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com> Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com> Co-authored-by: Miles Yucht <miles@databricks.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alex Ott <alexey.ott@databricks.com> Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com> Co-authored-by: Gautham Sunjay <gauthamsunjay17@gmail.com> Co-authored-by: guillesd <74136033+guillesd@users.noreply.github.com> Co-authored-by: Tanmay Rustagi <tanmay.rustagi@databricks.com> Co-authored-by: Tanmay Rustagi <88379306+tanmay-db@users.noreply.github.com> Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com> Co-authored-by: klus <lus.karol@gmail.com>
databricks · Oct 3, 2023 · 8641429 · 8641429
1 parent 6da1174
commit 8641429
Show file tree

Hide file tree

Showing 21 changed files with 1,270 additions and 552 deletions.
diff --git a/catalog/resource_metastore.go b/catalog/resource_metastore.go
@@ -5,24 +5,18 @@ import (
 	"log"
 	"strings"
 
+	"github.com/databricks/databricks-sdk-go"
+	"github.com/databricks/databricks-sdk-go/service/catalog"
 	"github.com/databricks/terraform-provider-databricks/common"
 	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
 	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/validation"
 )
 
-type MetastoresAPI struct {
-	client  *common.DatabricksClient
-	context context.Context
-}
-
-func NewMetastoresAPI(ctx context.Context, m any) MetastoresAPI {
-	return MetastoresAPI{m.(*common.DatabricksClient), context.WithValue(ctx, common.Api, common.API_2_1)}
-}
-
 type MetastoreInfo struct {
 	Name                                        string  `json:"name"`
 	StorageRoot                                 string  `json:"storage_root" tf:"force_new"`
 	DefaultDacID                                string  `json:"default_data_access_config_id,omitempty" tf:"suppress_diff"`
+	StorageRootCredentialId                     string  `json:"storage_root_credential_id,omitempty" tf:"suppress_diff"`
 	Owner                                       string  `json:"owner,omitempty" tf:"computed"`
 	MetastoreID                                 string  `json:"metastore_id,omitempty" tf:"computed"`
 	WorkspaceIDs                                []int64 `json:"workspace_ids,omitempty" tf:"computed"`
@@ -38,36 +32,9 @@ type MetastoreInfo struct {
 	DeltaSharingOrganizationName                string  `json:"delta_sharing_organization_name,omitempty"`
 }
 
-type CreateMetastore struct {
-	Name        string `json:"name"`
-	StorageRoot string `json:"storage_root"`
-}
-
-func (a MetastoresAPI) createMetastore(cm CreateMetastore) (mi MetastoreInfo, err error) {
-	err = a.client.Post(a.context, "/unity-catalog/metastores", cm, &mi)
-	return
-}
-
-func (a MetastoresAPI) getMetastore(id string) (mi MetastoreInfo, err error) {
-	err = a.client.Get(a.context, "/unity-catalog/metastores/"+id, nil, &mi)
-	return
-}
-
-func (a MetastoresAPI) updateMetastore(metastoreID string, update map[string]any) error {
-	return a.client.Patch(a.context, "/unity-catalog/metastores/"+metastoreID, update)
-}
-
-func (a MetastoresAPI) deleteMetastore(id string, force bool) error {
-	return a.client.Delete(a.context, "/unity-catalog/metastores/"+id, map[string]any{
-		"force": force,
-	})
-}
-
 func ResourceMetastore() *schema.Resource {
 	s := common.StructToSchema(MetastoreInfo{},
 		func(m map[string]*schema.Schema) map[string]*schema.Schema {
-			delete(m, "metastore_id")
-			delete(m, "workspace_ids") // todo: bring it back when it works
 			m["force_destroy"] = &schema.Schema{
 				Type:     schema.TypeBool,
 				Optional: true,
@@ -84,32 +51,89 @@ func ResourceMetastore() *schema.Resource {
 			}
 			return m
 		})
-	update := updateFunctionFactory("/unity-catalog/metastores", []string{"owner", "name", "delta_sharing_scope",
-		"delta_sharing_recipient_token_lifetime_in_seconds", "delta_sharing_organization_name"})
 
 	return common.Resource{
 		Schema: s,
 		Create: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
-			var create CreateMetastore
+			var create catalog.CreateMetastore
+			var update catalog.UpdateMetastore
 			common.DataToStructPointer(d, s, &create)
-			mi, err := NewMetastoresAPI(ctx, c).createMetastore(create)
-			if err != nil {
-				return err
-			}
-			d.SetId(mi.MetastoreID)
-			return update(ctx, d, c)
+			common.DataToStructPointer(d, s, &update)
+			return c.AccountOrWorkspaceRequest(func(acc *databricks.AccountClient) error {
+				mi, err := acc.Metastores.Create(ctx,
+					catalog.AccountsCreateMetastore{
+						MetastoreInfo: &create,
+					})
+				if err != nil {
+					return err
+				}
+				d.SetId(mi.MetastoreInfo.MetastoreId)
+				_, err = acc.Metastores.Update(ctx, catalog.AccountsUpdateMetastore{
+					MetastoreId:   mi.MetastoreInfo.MetastoreId,
+					MetastoreInfo: &update,
+				})
+				if err != nil {
+					return err
+				}
+				return nil
+			}, func(w *databricks.WorkspaceClient) error {
+				mi, err := w.Metastores.Create(ctx, create)
+				if err != nil {
+					return err
+				}
+				d.SetId(mi.MetastoreId)
+				update.Id = mi.MetastoreId
+				_, err = w.Metastores.Update(ctx, update)
+				if err != nil {
+					return err
+				}
+				return nil
+			})
 		},
 		Read: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
-			mi, err := NewMetastoresAPI(ctx, c).getMetastore(d.Id())
-			if err != nil {
-				return err
-			}
-			return common.StructToData(mi, s, d)
+			return c.AccountOrWorkspaceRequest(func(acc *databricks.AccountClient) error {
+				mi, err := acc.Metastores.GetByMetastoreId(ctx, d.Id())
+				if err != nil {
+					return err
+				}
+				return common.StructToData(mi, s, d)
+			}, func(w *databricks.WorkspaceClient) error {
+				mi, err := w.Metastores.GetById(ctx, d.Id())
+				if err != nil {
+					return err
+				}
+				return common.StructToData(mi, s, d)
+			})
+		},
+		Update: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
+			var update catalog.UpdateMetastore
+			common.DataToStructPointer(d, s, &update)
+
+			return c.AccountOrWorkspaceRequest(func(acc *databricks.AccountClient) error {
+				_, err := acc.Metastores.Update(ctx, catalog.AccountsUpdateMetastore{
+					MetastoreId:   d.Id(),
+					MetastoreInfo: &update,
+				})
+				if err != nil {
+					return err
+				}
+				return nil
+			}, func(w *databricks.WorkspaceClient) error {
+				update.Id = d.Id()
+				_, err := w.Metastores.Update(ctx, update)
+				if err != nil {
+					return err
+				}
+				return nil
+			})
 		},
-		Update: update,
 		Delete: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
 			force := d.Get("force_destroy").(bool)
-			return NewMetastoresAPI(ctx, c).deleteMetastore(d.Id(), force)
+			return c.AccountOrWorkspaceRequest(func(acc *databricks.AccountClient) error {
+				return acc.Metastores.Delete(ctx, catalog.DeleteAccountMetastoreRequest{Force: force, MetastoreId: d.Id()})
+			}, func(w *databricks.WorkspaceClient) error {
+				return w.Metastores.Delete(ctx, catalog.DeleteMetastoreRequest{Force: force, Id: d.Id()})
+			})
 		},
 	}.ToResource()
 }
diff --git a/catalog/resource_metastore_assignment.go b/catalog/resource_metastore_assignment.go
@@ -2,53 +2,18 @@ package catalog
 
 import (
 	"context"
-	"fmt"
+	"strconv"
 
+	"github.com/databricks/databricks-sdk-go"
+	"github.com/databricks/databricks-sdk-go/service/catalog"
 	"github.com/databricks/terraform-provider-databricks/common"
 	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
 )
 
-type MetastoreAssignmentAPI struct {
-	client  *common.DatabricksClient
-	context context.Context
-}
-
-func NewMetastoreAssignmentAPI(ctx context.Context, m any) MetastoreAssignmentAPI {
-	return MetastoreAssignmentAPI{m.(*common.DatabricksClient), context.WithValue(ctx, common.Api, common.API_2_1)}
-}
-
-type MetastoreAssignment struct {
-	WorkspaceID        int64  `json:"workspace_id" tf:"force_new"`
-	MetastoreID        string `json:"metastore_id"`
-	DefaultCatalogName string `json:"default_catalog_name,omitempty" tf:"default:hive_metastore"`
-}
-
-func (a MetastoreAssignmentAPI) createMetastoreAssignment(ma MetastoreAssignment) error {
-	path := fmt.Sprintf("/unity-catalog/workspaces/%d/metastore", ma.WorkspaceID)
-	return a.client.Put(a.context, path, ma)
-}
-
-func (a MetastoreAssignmentAPI) updateMetastoreAssignment(ma MetastoreAssignment) error {
-	path := fmt.Sprintf("/unity-catalog/workspaces/%d/metastore", ma.WorkspaceID)
-	return a.client.Patch(a.context, path, ma)
-}
-
-func (a MetastoreAssignmentAPI) getAssignedMetastoreID() (string, error) {
-	var mi MetastoreInfo
-	err := a.client.Get(a.context, "/unity-catalog/metastore_summary", nil, &mi)
-	return mi.MetastoreID, err
-}
-
-func (a MetastoreAssignmentAPI) deleteMetastoreAssignment(workspaceID, metastoreID string) error {
-	path := fmt.Sprintf("/unity-catalog/workspaces/%s/metastore", workspaceID)
-	return a.client.Delete(a.context, path, map[string]string{
-		"metastore_id": metastoreID,
-	})
-}
-
 func ResourceMetastoreAssignment() *schema.Resource {
-	s := common.StructToSchema(MetastoreAssignment{},
+	s := common.StructToSchema(catalog.MetastoreAssignment{},
 		func(m map[string]*schema.Schema) map[string]*schema.Schema {
+			m["default_catalog_name"].Default = "hive_metastore"
 			return m
 		})
 	pi := common.NewPairID("workspace_id", "metastore_id").Schema(
@@ -58,30 +23,95 @@ func ResourceMetastoreAssignment() *schema.Resource {
 	return common.Resource{
 		Schema: s,
 		Create: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
-			var ma MetastoreAssignment
-			common.DataToStructPointer(d, s, &ma)
-			if err := NewMetastoreAssignmentAPI(ctx, c).createMetastoreAssignment(ma); err != nil {
-				return err
-			}
-			pi.Pack(d)
-			return nil
+			workspaceId := int64(d.Get("workspace_id").(int))
+			metastoreId := d.Get("metastore_id").(string)
+			var create catalog.CreateMetastoreAssignment
+			common.DataToStructPointer(d, s, &create)
+			create.WorkspaceId = workspaceId
+
+			return c.AccountOrWorkspaceRequest(func(acc *databricks.AccountClient) error {
+				err := acc.MetastoreAssignments.Create(ctx,
+					catalog.AccountsCreateMetastoreAssignment{
+						WorkspaceId:         workspaceId,
+						MetastoreId:         metastoreId,
+						MetastoreAssignment: &create,
+					})
+				if err != nil {
+					return err
+				}
+				pi.Pack(d)
+				return nil
+			}, func(w *databricks.WorkspaceClient) error {
+				err := w.Metastores.Assign(ctx, create)
+				if err != nil {
+					return err
+				}
+				pi.Pack(d)
+				return nil
+			})
 		},
 		Read: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
-			metastoreID, err := NewMetastoreAssignmentAPI(ctx, c).getAssignedMetastoreID()
-			d.Set("metastore_id", metastoreID)
-			return err
+			first, _, err := pi.Unpack(d)
+			if err != nil {
+				return err
+			}
+			workspaceId, err := strconv.ParseInt(first, 10, 64)
+			if err != nil {
+				return err
+			}
+
+			return c.AccountOrWorkspaceRequest(func(acc *databricks.AccountClient) error {
+				ma, err := acc.MetastoreAssignments.GetByWorkspaceId(ctx, workspaceId)
+				if err != nil {
+					return err
+				}
+				return common.StructToData(ma, s, d)
+			}, func(w *databricks.WorkspaceClient) error {
+				//this only works when managing the metastore assigned to the current workspace.
+				//plus we don't know the workspace we're logged into.
+				ma, err := w.Metastores.Current(ctx)
+				if err != nil {
+					return err
+				}
+				d.Set("metastore_id", ma.MetastoreId)
+				return nil
+			})
 		},
 		Update: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
-			var ma MetastoreAssignment
-			common.DataToStructPointer(d, s, &ma)
-			return NewMetastoreAssignmentAPI(ctx, c).updateMetastoreAssignment(ma)
+			workspaceId := int64(d.Get("workspace_id").(int))
+			metastoreId := d.Get("metastore_id").(string)
+			var update catalog.UpdateMetastoreAssignment
+			common.DataToStructPointer(d, s, &update)
+			update.WorkspaceId = workspaceId
+
+			return c.AccountOrWorkspaceRequest(func(acc *databricks.AccountClient) error {
+				return acc.MetastoreAssignments.Update(ctx,
+					catalog.AccountsUpdateMetastoreAssignment{
+						WorkspaceId:         workspaceId,
+						MetastoreId:         metastoreId,
+						MetastoreAssignment: &update,
+					})
+			}, func(w *databricks.WorkspaceClient) error {
+				return w.Metastores.UpdateAssignment(ctx, update)
+			})
 		},
 		Delete: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
-			workspaceID, metastoreID, err := pi.Unpack(d)
+			first, metastoreId, err := pi.Unpack(d)
+			if err != nil {
+				return err
+			}
+			workspaceId, err := strconv.ParseInt(first, 10, 64)
 			if err != nil {
 				return err
 			}
-			return NewMetastoreAssignmentAPI(ctx, c).deleteMetastoreAssignment(workspaceID, metastoreID)
+			return c.AccountOrWorkspaceRequest(func(acc *databricks.AccountClient) error {
+				return acc.MetastoreAssignments.DeleteByWorkspaceIdAndMetastoreId(ctx, workspaceId, metastoreId)
+			}, func(w *databricks.WorkspaceClient) error {
+				return w.Metastores.Unassign(ctx, catalog.UnassignRequest{
+					MetastoreId: metastoreId,
+					WorkspaceId: workspaceId,
+				})
+			})
 		},
 	}.ToResource()
 }