Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for Unity Catalog databricks_metastores data source #2017

Merged
merged 49 commits into from
Jul 19, 2023
Merged
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
12b841e
add documentation for databricks_metastores data source
guillesd Feb 17, 2023
8b10dab
add API endpoint for listing metastores
guillesd Feb 17, 2023
9082709
add metastores data resource
guillesd Feb 17, 2023
921c288
add test for metastores data source
guillesd Feb 17, 2023
f0d146f
add metastores datasource to resource mapping
guillesd Feb 18, 2023
22825c6
fix reference to wrong resource docs
guillesd Feb 20, 2023
399d76d
add a Metastores struct for the response of the API, use this in the …
guillesd Feb 20, 2023
4eaa8c2
update terraform specific object attributes
guillesd Feb 20, 2023
857d28e
add new data test
guillesd Feb 20, 2023
e345f2f
remove slice_set property from MetastoreData
guillesd Feb 20, 2023
7b4f4a2
Merge branch 'master' into feature/add_metastore_datasource
guillesd Feb 28, 2023
1473c0e
Merge branch 'master' into feature/add_metastore_datasource
guillesd Mar 17, 2023
53cbf83
use databricks-go-sdk for data_metastore.go
guillesd Mar 17, 2023
5601f47
removed listMetastores endpoint since it's unused
guillesd Mar 17, 2023
3dfbdbc
make sure tests also use the unitycatalog.MetastoreInfo from the sdk
guillesd Mar 17, 2023
1beb87a
remove redundant resource
guillesd Mar 17, 2023
70f9fa4
Merge branch 'master' into feature/add_metastore_datasource
guillesd Jun 9, 2023
aa70996
Merge branch 'master' of github.com:databricks/terraform-provider-dat…
tanmay-db Jun 13, 2023
76c5d2b
test -dev
tanmay-db Jun 13, 2023
6c111e6
fix
tanmay-db Jun 21, 2023
c04d6c5
fmt
tanmay-db Jun 21, 2023
0f4c815
Merge remote-tracking branch 'databricks/master' into feature/add_met…
tanmay-db Jun 21, 2023
fe9bf12
cleanup
tanmay-db Jun 21, 2023
d469bdf
Added AccountClient to DatabricksClient and AccountData
tanmay-db Jun 26, 2023
3853b84
Merge branch 'AccountData' into feature/add_metastore_datasource-dev
tanmay-db Jun 26, 2023
014dd6f
upd
tanmay-db Jun 26, 2023
a77a717
cleanup
tanmay-db Jun 26, 2023
ee3e70f
accountLevel
tanmay-db Jun 26, 2023
c9a3f4f
Merge branch 'master' of github.com:databricks/terraform-provider-dat…
tanmay-db Jun 29, 2023
ba97743
upd
tanmay-db Jun 29, 2023
cf89129
Merge branch 'AccountData' into feature/add_metastore_datasource-dev
tanmay-db Jun 29, 2023
aaad551
add example
tanmay-db Jun 29, 2023
85da2fe
Merge branch 'AccountData' into feature/add_metastore_datasource-dev
tanmay-db Jun 29, 2023
df55236
list
tanmay-db Jun 29, 2023
c78d412
cleanup
tanmay-db Jun 29, 2023
8d27960
docs
tanmay-db Jun 29, 2023
a3efb4c
remove dead code
tanmay-db Jun 29, 2023
1f5142f
upd to master
tanmay-db Jul 3, 2023
eb40c81
wip
tanmay-db Jul 3, 2023
fd9408d
Merge branch 'master' into feature/add_metastore_datasource
nkvuong Jul 7, 2023
d6bf74d
use maps
tanmay-db Jul 18, 2023
3e0eb03
Merge remote-tracking branch 'databricks/master' into feature/add_met…
tanmay-db Jul 18, 2023
c8f790f
Merge branch 'feature/add_metastore_datasource' of github.com:guilles…
tanmay-db Jul 18, 2023
d173572
upd
tanmay-db Jul 18, 2023
e97710f
cleanup
tanmay-db Jul 18, 2023
32f80aa
comments
tanmay-db Jul 19, 2023
95c0b94
Merge branch 'master' of github.com:databricks/terraform-provider-dat…
tanmay-db Jul 19, 2023
c81d646
-
tanmay-db Jul 19, 2023
6dfade4
remove redundant test
tanmay-db Jul 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions catalog/data_metastores.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package catalog

import (
"context"

"github.com/databricks/databricks-sdk-go"
"github.com/databricks/terraform-provider-databricks/common"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)

func DataSourceMetastores() *schema.Resource {
return common.AccountData(func(ctx context.Context, data *MetastoresData, acc *databricks.AccountClient) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metastores, err := acc.Metastores.List(ctx)
if err != nil {
return err
}
for _, v := range metastores.Metastores {
data.Ids = append(data.Ids, v.MetastoreId)
}
return nil
})
}
92 changes: 92 additions & 0 deletions catalog/data_metastores_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
package catalog

import (
"testing"

"github.com/databricks/databricks-sdk-go/service/catalog"
"github.com/databricks/terraform-provider-databricks/qa"
)

func TestMetastoresData(t *testing.T) {
qa.ResourceFixture{
Fixtures: []qa.HTTPFixture{
{
Method: "GET",
Resource: "/api/2.0/accounts/testaccount/metastores",
tanmay-db marked this conversation as resolved.
Show resolved Hide resolved
Response: catalog.ListMetastoresResponse{
Metastores: []catalog.MetastoreInfo{
{
Name: "a",
StorageRoot: "",
DefaultDataAccessConfigId: "sth",
Owner: "John.Doe@example.com",
MetastoreId: "abc",
Region: "",
Cloud: "",
GlobalMetastoreId: "",
CreatedAt: 0,
CreatedBy: "",
UpdatedAt: 0,
UpdatedBy: "",
DeltaSharingScope: "",
DeltaSharingRecipientTokenLifetimeInSeconds: 0,
DeltaSharingOrganizationName: "",
},
},
},
},
},
Resource: DataSourceMetastores(),
Read: true,
NonWritable: true,
ID: "_",
AccountID: "testaccount",
}.ApplyNoError(t)
guillesd marked this conversation as resolved.
Show resolved Hide resolved
}

func TestMetastoresDataContainsName(t *testing.T) {
qa.ResourceFixture{
Fixtures: []qa.HTTPFixture{
{
Method: "GET",
Resource: "/api/2.0/accounts/testaccount/metastores",
Response: catalog.ListMetastoresResponse{
Metastores: []catalog.MetastoreInfo{
{
Name: "a",
StorageRoot: "abc",
DefaultDataAccessConfigId: "sth",
Owner: "John.Doe@example.com",
MetastoreId: "abc",
},
{
Name: "b",
StorageRoot: "dcw",
DefaultDataAccessConfigId: "sth",
Owner: "John.Doe@example.com",
MetastoreId: "ded",
},
},
},
},
},
Resource: DataSourceMetastores(),
Read: true,
NonWritable: true,
ID: "_",
AccountID: "testaccount",
}.ApplyAndExpectData(t, map[string]any{
"ids": []string{"abc", "ded"},
})
}

func TestMetastoresData_Error(t *testing.T) {
qa.ResourceFixture{
Fixtures: qa.HTTPFailures,
Resource: DataSourceMetastores(),
Read: true,
NonWritable: true,
ID: "_",
AccountID: "_",
}.ExpectError(t, "I'm a teapot")
}
9 changes: 4 additions & 5 deletions catalog/resource_metastore.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,15 @@ type MetastoreInfo struct {
DeltaSharingOrganizationName string `json:"delta_sharing_organization_name,omitempty"`
}

type MetastoresData struct {
Ids []string `json:"ids,omitempty" tf:"computed,slice_set"`
}

type CreateMetastore struct {
Name string `json:"name"`
StorageRoot string `json:"storage_root"`
}

// func (a MetastoresAPI) listMetastores() (mis []MetastoreInfo, err error) {
// err = a.client.Get(a.context, "/unity-catalog/metastores", nil, &mis)
// return
// }

Comment on lines -46 to -50
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove dead code, we already have crud

func (a MetastoresAPI) createMetastore(cm CreateMetastore) (mi MetastoreInfo, err error) {
err = a.client.Post(a.context, "/unity-catalog/metastores", cm, &mi)
return
Expand Down
33 changes: 33 additions & 0 deletions docs/data-sources/metastores.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
subcategory: "Unity Catalog"
---
# databricks_metastores Data Source

-> **Note** If you have a fully automated setup with workspaces created by [databricks_mws_workspaces](../resources/mws_workspaces.md) or [azurerm_databricks_workspace](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_workspace), please make sure to add [depends_on attribute](../index.md#data-resources-and-authentication-is-not-configured-errors) in order to prevent _authentication is not configured for provider_ errors.

Retrieves a list of ids of [databricks_metastore](../resources/metastore.md) objects, that were created by Terraform or manually, so that special handling could be applied.

## Example Usage

Listing ids of all catalogs:

```hcl
data "databricks_metastores" "all" {}

output "all_metastores" {
value = data.databricks_metastores.all
}
```

## Attribute Reference

This data source exports the following attributes:

* `metastores` - list of ids of [databricks_metastore](../resources/metastore.md)

## Related Resources

The following resources are used in the same context:

* [databricks_metastore](../resources/metastore.md) to manage Metastores within Unity Catalog.
* [databricks_catalog](../resources/catalog.md) to manage catalogs within Unity Catalog.
16 changes: 16 additions & 0 deletions internal/acceptance/data_metastores_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package acceptance

import (
"testing"
)

func TestUcAccDataSourceMetastore(t *testing.T) {
accountLevel(t, step{
Template: `
data "databricks_metastores" "this" {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a reasonable integration test for the "no metastores" case. Can we add an integration test for the "one or more metastores case"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mgyucht not sure how easy it is to test both scenarios in a single account, unless we add a filter option to this data source

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are going to add another data source for specific metastore that contains the full information. Creating a specific metastore and getting it's details will be part of integration test for that data source. Marking this as non blocking for this data source since we have permanent metastores for our account. Also verified locally looking at the results of using the resource.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check has been added in the test


output "all_metastores" {
value = data.databricks_metastores.this
}`,
})
}
1 change: 1 addition & 0 deletions provider/provider.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ func DatabricksProvider() *schema.Provider {
"databricks_instance_pool": pools.DataSourceInstancePool(),
"databricks_jobs": jobs.DataSourceJobs(),
"databricks_job": jobs.DataSourceJob(),
"databricks_metastores": catalog.DataSourceMetastores(),
tanmay-db marked this conversation as resolved.
Show resolved Hide resolved
"databricks_mws_credentials": mws.DataSourceMwsCredentials(),
"databricks_mws_workspaces": mws.DataSourceMwsWorkspaces(),
"databricks_node_type": clusters.DataSourceNodeType(),
Expand Down