Skip to content

Commit

Permalink
Merge branch 'master' into patch-2
Browse files Browse the repository at this point in the history
  • Loading branch information
fuselessmatt authored Jan 2, 2023
2 parents f061b77 + ab3df33 commit a576eba
Show file tree
Hide file tree
Showing 56 changed files with 1,275 additions and 354 deletions.
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,40 @@
# Version changelog

### 1.7.0

* Added support for Unity Catalog on GCP through `gcp_service_account_key` ([#1836](https://github.com/databricks/terraform-provider-databricks/pull/1836)).
* Added support for Jupyter (`.ipynb`) files in [databricks_notebook](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/databricks_notebook) resource ([#1801](https://github.com/databricks/terraform-provider-databricks/pull/1801)).
* Added [databricks_cluster_policy](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/cluster_policy) data source ([#1792](https://github.com/databricks/terraform-provider-databricks/pull/1792)).
* Added `catalog` parameter to `dbt_task` in [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource ([#1774](https://github.com/databricks/terraform-provider-databricks/pull/1774)).
* Added `home` and `repos` attributes to [databricks_user](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/user) and [databricks_service_principal](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/service_principal) ([#1791](https://github.com/databricks/terraform-provider-databricks/pull/1791)).
* Added `storage_root` parameter in [databricks_catalog](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/catalog) and [databricks_schema](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/schema) resources ([#1784](https://github.com/databricks/terraform-provider-databricks/pull/1784)).
* Added exporting of [databricks_service_principal](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/service_principal) resource ([#1811](https://github.com/databricks/terraform-provider-databricks/pull/1811)).
* Added notebook `source` parameter for [databricks_jobs](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/jobs) data ([#1837](https://github.com/databricks/terraform-provider-databricks/pull/1837)).
* Added support for init scripts on ADLS in [databricks_cluster](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster) and related resources ([#1845](https://github.com/databricks/terraform-provider-databricks/pull/1845)).
* Added `definiton` attribute to [databricks_cluster_policy](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/cluster_policy) data ([#1834](https://github.com/databricks/terraform-provider-databricks/pull/1834)).
* Added validation for preview account API paths ([#1795](https://github.com/databricks/terraform-provider-databricks/pull/1795)).
* Made `lifetime_seconds` & `comment` optional for [databricks_obo_token](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/obo_token) ([#1844](https://github.com/databricks/terraform-provider-databricks/pull/1844)).
* Force replace when changing `display_name` of [databricks_service_principal](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/service_principal) ([#1783](https://github.com/databricks/terraform-provider-databricks/pull/1783)).
* We can now export `admin` and `users` groups as [databricks_group](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group) data, not resource ([#1721](https://github.com/databricks/terraform-provider-databricks/pull/1721)).
* We can now export predefined [databricks_cluster_policy](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster_policy) as data sources ([#1815](https://github.com/databricks/terraform-provider-databricks/pull/1815)).
* We can now export [databricks_sql_global_config](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/sql_global_config) resource ([#1806](https://github.com/databricks/terraform-provider-databricks/pull/1806)).
* Fixed reference to [databricks_cluster_policy](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster_policy) in [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) during export ([#1813](https://github.com/databricks/terraform-provider-databricks/pull/1813)).
* We can now explicitly normalize variable names during export ([#1835](https://github.com/databricks/terraform-provider-databricks/pull/1835)).
* We can now fail fast in `import.sh` when can't import resource ([#1814](https://github.com/databricks/terraform-provider-databricks/pull/1814)).
* Fixed multidomain [databricks_user](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/user) in exporter ([#1722](https://github.com/databricks/terraform-provider-databricks/pull/1722)).
* Fixed missing attributes from [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/job) data source ([#1798](https://github.com/databricks/terraform-provider-databricks/pull/1798)).
* Suppressed `network` diff for GCP workspaces with Databricks-managed VPC ([#1847](https://github.com/databricks/terraform-provider-databricks/pull/1847)).
* Improved export of DBSQL objects ([#1788](https://github.com/databricks/terraform-provider-databricks/pull/1788)).
* Documented `webhook_notifications` for [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job) resource ([#1842](https://github.com/databricks/terraform-provider-databricks/pull/1842)).
* Clarified [databricks_job](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job), `databricks_mws_*`, [databricks_cluster](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster) docs ([#1796](https://github.com/databricks/terraform-provider-databricks/pull/1796)).

Updated dependency versions:

* Bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.4.3 ([#1817](https://github.com/databricks/terraform-provider-databricks/pull/1817)).
* Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.24.0 to 2.24.1 ([#1761](https://github.com/databricks/terraform-provider-databricks/pull/1761)).
* Bump golang.org/x/mod from 0.6.0 to 0.7.0 ([#1760](https://github.com/databricks/terraform-provider-databricks/pull/1760)).
* Bump google.golang.org/api from 0.102.0 to 0.104.0 ([#1758](https://github.com/databricks/terraform-provider-databricks/pull/1758), [#1839](https://github.com/databricks/terraform-provider-databricks/pull/1839)).

### 1.6.5

* Added `query_plan` attribute to `databricks_sql_visualization` ([#1752](https://github.com/databricks/terraform-provider-databricks/pull/1752)).
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ terraform {
required_providers {
databricks = {
source = "databricks/databricks"
version = "1.6.5"
version = "1.7.0"
}
}
}
Expand Down
28 changes: 28 additions & 0 deletions catalog/acceptance/provider_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package acceptance

import (
"testing"

"github.com/databricks/terraform-provider-databricks/internal/acceptance"
"github.com/databricks/terraform-provider-databricks/qa"
)

func TestUcAccCreateProviderDb2Open(t *testing.T) {
qa.RequireCloudEnv(t, "ucws")
acceptance.Test(t, []acceptance.Step{
{
Template: `
resource "databricks_provider" "this" {
name = "terraform-test-provider"
comment = "made by terraform"
authentication_type = "TOKEN"
recipient_profile_str = jsonencode({
"shareCredentialsVersion":1,
"bearerToken":"dapiabcdefghijklmonpqrstuvwxyz",
"endpoint":"https://sharing.delta.io/delta-sharing/"
}
)
}`,
},
})
}
27 changes: 22 additions & 5 deletions catalog/resource_metastore_data_access.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,22 +24,31 @@ type AwsIamRole struct {
type AzureServicePrincipal struct {
DirectoryID string `json:"directory_id"`
ApplicationID string `json:"application_id"`
ClientSecret string `json:"client_secret"`
ClientSecret string `json:"client_secret" tf:"sensitive"`
}

type AzureManagedIdentity struct {
AccessConnectorID string `json:"access_connector_id"`
}

type GcpServiceAccountKey struct {
Email string `json:"email"`
PrivateKeyId string `json:"private_key_id"`
PrivateKey string `json:"private_key" tf:"sensitive"`
}

type DataAccessConfiguration struct {
ID string `json:"id,omitempty" tf:"computed"`
Name string `json:"name"`
ConfigurationType string `json:"configuration_type,omitempty" tf:"computed"`
Aws *AwsIamRole `json:"aws_iam_role,omitempty" tf:"group:access"`
Azure *AzureServicePrincipal `json:"azure_service_principal,omitempty" tf:"group:access"`
AzMI *AzureManagedIdentity `json:"azure_managed_identity,omitempty" tf:"group:access"`
GcpSAKey *GcpServiceAccountKey `json:"gcp_service_account_key,omitempty" tf:"group:access"`
}

var alofCred = []string{"aws_iam_role", "azure_service_principal", "azure_managed_identity", "gcp_service_account_key"}

func (a DataAccessConfigurationsAPI) Create(metastoreID string, dac *DataAccessConfiguration) error {
path := fmt.Sprintf("/unity-catalog/metastores/%s/data-access-configurations", metastoreID)
return a.client.Post(a.context, path, dac, dac)
Expand All @@ -56,6 +65,11 @@ func (a DataAccessConfigurationsAPI) Delete(metastoreID, dacID string) error {
return a.client.Delete(a.context, path, nil)
}

func SuppressGcpSAKeyDiff(k, old, new string, d *schema.ResourceData) bool {
//ignore changes in private_key
return !d.HasChanges("gcp_service_account_key.0.email", "gcp_service_account_key.0.private_key_id")
}

func ResourceMetastoreDataAccess() *schema.Resource {
s := common.StructToSchema(DataAccessConfiguration{},
func(m map[string]*schema.Schema) map[string]*schema.Schema {
Expand All @@ -70,10 +84,13 @@ func ResourceMetastoreDataAccess() *schema.Resource {
Type: schema.TypeBool,
Optional: true,
}
alof := []string{"aws_iam_role", "azure_service_principal", "azure_managed_identity"}
m["aws_iam_role"].AtLeastOneOf = alof
m["azure_service_principal"].AtLeastOneOf = alof
m["azure_managed_identity"].AtLeastOneOf = alof
m["aws_iam_role"].AtLeastOneOf = alofCred
m["azure_service_principal"].AtLeastOneOf = alofCred
m["azure_managed_identity"].AtLeastOneOf = alofCred
m["gcp_service_account_key"].AtLeastOneOf = alofCred

// suppress changes for private_key
m["gcp_service_account_key"].DiffSuppressFunc = SuppressGcpSAKeyDiff
return m
})
p := common.NewPairID("metastore_id", "id")
Expand Down
59 changes: 59 additions & 0 deletions catalog/resource_metastore_data_access_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,62 @@ func TestCreateDacWithAzMI(t *testing.T) {
`,
}.ApplyNoError(t)
}

func TestCreateDacWithGcpSA(t *testing.T) {
qa.ResourceFixture{
Fixtures: []qa.HTTPFixture{
{
Method: "POST",
Resource: "/api/2.1/unity-catalog/metastores/abc/data-access-configurations",
ExpectedRequest: DataAccessConfiguration{
Name: "bcd",
GcpSAKey: &GcpServiceAccountKey{
Email: "a@example.com",
PrivateKeyId: "b",
PrivateKey: "abcdefg",
},
},
Response: DataAccessConfiguration{
ID: "efg",
},
},
{
Method: "PATCH",
Resource: "/api/2.1/unity-catalog/metastores/abc",
ExpectedRequest: map[string]any{
"default_data_access_config_id": "efg",
},
},
{
Method: "GET",
Resource: "/api/2.1/unity-catalog/metastores/abc/data-access-configurations/efg",
Response: DataAccessConfiguration{
Name: "bcd",
GcpSAKey: &GcpServiceAccountKey{
Email: "a@example.com",
PrivateKeyId: "b",
},
},
},
{
Method: "GET",
Resource: "/api/2.1/unity-catalog/metastores/abc",
Response: MetastoreInfo{
DefaultDacID: "efg",
},
},
},
Create: true,
Resource: ResourceMetastoreDataAccess(),
HCL: `
metastore_id = "abc"
name = "bcd"
is_default = true
gcp_service_account_key {
email = "a@example.com"
private_key_id = "b"
private_key = "abcdefg"
}
`,
}.ApplyNoError(t)
}
91 changes: 91 additions & 0 deletions catalog/resource_provider.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
package catalog

import (
"context"

"github.com/databricks/terraform-provider-databricks/common"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/validation"
)

type ProvidersAPI struct {
client *common.DatabricksClient
context context.Context
}

func NewProvidersAPI(ctx context.Context, m interface{}) ProvidersAPI {
return ProvidersAPI{m.(*common.DatabricksClient), ctx}
}

type ProviderInfo struct {
Name string `json:"name" tf:"force_new"`
Comment string `json:"comment,omitempty"`
AuthenticationType string `json:"authentication_type"`
RecipientProfileStr string `json:"recipient_profile_str" tf:"sensitive"`
}

type Providers struct {
Providers []ProviderInfo `json:"providers"`
}

func (a ProvidersAPI) createProvider(ci *ProviderInfo) error {
return a.client.Post(a.context, "/unity-catalog/providers", ci, ci)
}

func (a ProvidersAPI) getProvider(name string) (ci ProviderInfo, err error) {
err = a.client.Get(a.context, "/unity-catalog/providers/"+name, nil, &ci)
return
}

func (a ProvidersAPI) deleteProvider(name string) error {
return a.client.Delete(a.context, "/unity-catalog/providers/"+name, nil)
}

func (a ProvidersAPI) updateProvider(ci *ProviderInfo) error {
patch := struct {
Comment string `json:"comment"`
}{
Comment: ci.Comment,
}
return a.client.Patch(a.context, "/unity-catalog/providers/"+ci.Name, patch)
}

func ResourceProvider() *schema.Resource {
providerSchema := common.StructToSchema(ProviderInfo{}, func(m map[string]*schema.Schema) map[string]*schema.Schema {
m["authentication_type"].ValidateFunc = validation.StringInSlice([]string{"TOKEN"}, false)
return m
})

providerSchemaForRead := map[string]*schema.Schema{
"name": providerSchema["name"],
"comment": providerSchema["comment"],
"authentication_type": providerSchema["authentication_type"],
}
return common.Resource{
Schema: providerSchema,
Create: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
var ri ProviderInfo
common.DataToStructPointer(d, providerSchema, &ri)
if err := NewProvidersAPI(ctx, c).createProvider(&ri); err != nil {
return err
}
d.SetId(ri.Name)
return nil
},
Read: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
ri, err := NewProvidersAPI(ctx, c).getProvider(d.Id())
if err != nil {
return err
}
return common.StructToData(ri, providerSchemaForRead, d)
},
Update: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
var ri ProviderInfo
common.DataToStructPointer(d, providerSchema, &ri)
return NewProvidersAPI(ctx, c).updateProvider(&ri)
},
Delete: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
return NewProvidersAPI(ctx, c).deleteProvider(d.Id())
},
}.ToResource()
}
66 changes: 66 additions & 0 deletions catalog/resource_provider_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package catalog

import (
"testing"

"github.com/databricks/terraform-provider-databricks/qa"
)

func TestProviderCornerCases(t *testing.T) {
qa.ResourceCornerCases(t, ResourceProvider())
}

func TestCreateProvider(t *testing.T) {
qa.ResourceFixture{
Fixtures: []qa.HTTPFixture{
{
Method: "POST",
Resource: "/api/2.0/unity-catalog/providers",
ExpectedRequest: ProviderInfo{
Name: "a",
Comment: "b",
AuthenticationType: "TOKEN",
RecipientProfileStr: "c",
},
Response: ProviderInfo{
Name: "a",
},
},
{
Method: "GET",
Resource: "/api/2.0/unity-catalog/providers/a",
Response: ProviderInfo{
Name: "a",
Comment: "b",
AuthenticationType: "TOKEN",
RecipientProfileStr: "c",
},
},
},
Resource: ResourceProvider(),
Create: true,
HCL: `
name = "a"
comment = "b"
authentication_type = "TOKEN"
recipient_profile_str = "c"
`,
}.ApplyNoError(t)
}

func TestCreateProvider_InvalidAuthType(t *testing.T) {
qa.ResourceFixture{
Fixtures: []qa.HTTPFixture{},
Resource: ResourceProvider(),
Create: true,
HCL: `
name = "a"
comment = "b"
authentication_type = "temp"
recipient_profile_str = "{\"shareCredentialsVersion\":1,\"bearerToken\":\"a\",\"endpoint\":\"b\",\"expirationTime\":\"c\"}"
`,
}.ExpectError(t, "invalid config supplied. "+
"[authentication_type] expected authentication_type "+
"to be one of [TOKEN], got temp")

}
15 changes: 10 additions & 5 deletions catalog/resource_storage_credential.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ type StorageCredentialInfo struct {
Aws *AwsIamRole `json:"aws_iam_role,omitempty" tf:"group:access"`
Azure *AzureServicePrincipal `json:"azure_service_principal,omitempty" tf:"group:access"`
AzMI *AzureManagedIdentity `json:"azure_managed_identity,omitempty" tf:"group:access"`
GcpSAKey *GcpServiceAccountKey `json:"gcp_service_account_key,omitempty" tf:"group:access"`
MetastoreID string `json:"metastore_id,omitempty" tf:"computed"`
}

Expand All @@ -42,14 +43,18 @@ func (a StorageCredentialsAPI) delete(id string) error {
func ResourceStorageCredential() *schema.Resource {
s := common.StructToSchema(StorageCredentialInfo{},
func(m map[string]*schema.Schema) map[string]*schema.Schema {
alof := []string{"aws_iam_role", "azure_service_principal", "azure_managed_identity"}
m["aws_iam_role"].AtLeastOneOf = alof
m["azure_service_principal"].AtLeastOneOf = alof
m["azure_managed_identity"].AtLeastOneOf = alof
m["aws_iam_role"].AtLeastOneOf = alofCred
m["azure_service_principal"].AtLeastOneOf = alofCred
m["azure_managed_identity"].AtLeastOneOf = alofCred
m["gcp_service_account_key"].AtLeastOneOf = alofCred

// suppress changes for private_key
m["gcp_service_account_key"].DiffSuppressFunc = SuppressGcpSAKeyDiff

return m
})
update := updateFunctionFactory("/unity-catalog/storage-credentials", []string{
"owner", "comment", "aws_iam_role", "azure_service_principal", "azure_managed_identity"})
"owner", "comment", "aws_iam_role", "azure_service_principal", "azure_managed_identity", "gcp_service_account_key"})
return common.Resource{
Schema: s,
Create: func(ctx context.Context, d *schema.ResourceData, c *common.DatabricksClient) error {
Expand Down
Loading

0 comments on commit a576eba

Please sign in to comment.