Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Database resource v1 readiness #2834

Merged
merged 8 commits into from
May 29, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion MIGRATION_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,17 @@ This document is meant to help you migrate your Terraform config to the new newe
describe deprecations or breaking changes and help you to change your configuration to keep the same (or similar) behavior
across different versions.

## v0.91.0 ➞ v0.92.0
### snowflake_database new alternatives
sfc-gh-asawicki marked this conversation as resolved.
Show resolved Hide resolved
As part of the [preparation for v1](https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/ROADMAP.md#preparing-essential-ga-objects-for-the-provider-v1), we split up the database resource into multiple ones:
- Standard database (in progress)
- Shared database - can be used as `snowflake_shared_database` (used to create databases from externally defined shares)
- Secondary database - can be used as `snowflake_secondary_database` (used to create replicas of databases from external sources)
From now on, please migrate and use the new database resources for their unique use cases. For more information, see the documentation for those resources on the [Terraform Registry](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs).

The split was done (and will be done for several objects during the refactor) to simplify the resource on maintainability and usage level.
Its purpose was also to divide the resources by their specific purpose rather than cramping every use case of an object into one resource.

## v0.89.0 ➞ v0.90.0
### snowflake_table resource changes
#### *(behavior change)* Validation to column type added
Expand All @@ -23,7 +34,7 @@ resource "snowflake_tag_masking_policy_association" "name" {
masking_policy_id = snowflake_masking_policy.example_masking_policy.id
}
```

After
```terraform
resource "snowflake_tag_masking_policy_association" "name" {
Expand Down
96 changes: 96 additions & 0 deletions docs/resources/secondary_database.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
page_title: "snowflake_secondary_database Resource - terraform-provider-snowflake"
subcategory: ""
description: |-
A secondary database creates a replica of an existing primary database (i.e. a secondary database). For more information about database replication, see Introduction to database replication across multiple accounts https://docs.snowflake.com/en/user-guide/db-replication-intro.
---

# snowflake_secondary_database (Resource)

A secondary database creates a replica of an existing primary database (i.e. a secondary database). For more information about database replication, see [Introduction to database replication across multiple accounts](https://docs.snowflake.com/en/user-guide/db-replication-intro).

## Example Usage

```terraform
# 1. Preparing primary database
resource "snowflake_database" "primary" {
provider = primary_account # notice the provider fields
name = "database_name"
replication_configuration {
accounts = ["<secondary_account_organization_name>.<secondary_account_name>"]
ignore_edition_check = true
}
}

# 2. Creating secondary database
resource "snowflake_secondary_database" "test" {
provider = secondary_account
name = snowflake_database.primary.name # It's recommended to give a secondary database the same name as its primary database
as_replica_of = "<primary_account_organization_name>.<primary_account_name>.${snowflake_database.primary.name}"
is_transient = false

data_retention_time_in_days {
value = 10
}

max_data_extension_time_in_days {
value = 20
}

external_volume = "external_volume_name"
catalog = "catalog_name"
default_ddl_collation = "en_US"
sfc-gh-jmichalak marked this conversation as resolved.
Show resolved Hide resolved
log_level = "OFF"
trace_level = "OFF"
comment = "A secondary database"
}
```

<!-- schema generated by tfplugindocs -->
## Schema

### Required

- `as_replica_of` (String) A fully qualified path to a database to create a replica from. A fully qualified path follows the format of `"<organization_name>"."<account_name>"."<database_name>"`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just checking: this is the new recommended Snowflake way of identifying accounts, right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I changed to the documentation one <provider_account>.<share_name>, because we should be able to handle both identifiers. I'll modify one integration and acceptance test to show it's working.

Copy link
Collaborator Author

@sfc-gh-jcieslak sfc-gh-jcieslak May 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, we can easily support both formats on SDK level, but it's hard to support both ways in Terraform because in Read we have to assume one format. For now, I'll revert the acceptance test, because it's failing for account locator and maybe we should enforce the new format instead of supporting the old one(?). That's why originally I stayed with this description, I can revert the change to "<organization_name>"."<account_name>"."<database_name>".

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it would be best to stick with the SF recommended way. This is a change against the old behavior but it is a new resource after all. However, we should mention this change in the migration notes.

Copy link
Collaborator Author

@sfc-gh-jcieslak sfc-gh-jcieslak May 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I'll revert the format (in the next pr) and add a note in migration guide on expected external identifiers

- `name` (String) Specifies the identifier for the database; must be unique for your account. As a best practice for [Database Replication and Failover](https://docs.snowflake.com/en/user-guide/db-replication-intro), it is recommended to give each secondary database the same name as its primary database. This practice supports referencing fully-qualified objects (i.e. '<db>.<schema>.<object>') by other objects in the same database, such as querying a fully-qualified table name in a view. If a secondary database has a different name from the primary database, then these object references would break in the secondary database.

### Optional

- `catalog` (String) The database parameter that specifies the default catalog to use for Iceberg tables.
- `comment` (String) Specifies a comment for the database.
- `data_retention_time_in_days` (Block List, Max: 1) Specifies the number of days for which Time Travel actions (CLONE and UNDROP) can be performed on the database, as well as specifying the default Time Travel retention time for all schemas created in the database. For more details, see [Understanding & Using Time Travel](https://docs.snowflake.com/en/user-guide/data-time-travel). (see [below for nested schema](#nestedblock--data_retention_time_in_days))
- `default_ddl_collation` (String) Specifies a default collation specification for all schemas and tables added to the database. It can be overridden on schema or table level. For more information, see [collation specification](https://docs.snowflake.com/en/sql-reference/collation#label-collation-specification).
- `external_volume` (String) The database parameter that specifies the default external volume to use for Iceberg tables.
- `is_transient` (Boolean) Specifies the database as transient. Transient databases do not have a Fail-safe period so they do not incur additional storage costs once they leave Time Travel; however, this means they are also not protected by Fail-safe in the event of a data loss.
- `log_level` (String) Specifies the severity level of messages that should be ingested and made available in the active event table. Valid options are: [TRACE DEBUG INFO WARN ERROR FATAL OFF]. Messages at the specified level (and at more severe levels) are ingested. For more information, see [LOG_LEVEL](https://docs.snowflake.com/en/sql-reference/parameters.html#label-log-level).
- `max_data_extension_time_in_days` (Block List, Max: 1) Object parameter that specifies the maximum number of days for which Snowflake can extend the data retention period for tables in the database to prevent streams on the tables from becoming stale. For a detailed description of this parameter, see [MAX_DATA_EXTENSION_TIME_IN_DAYS](https://docs.snowflake.com/en/sql-reference/parameters.html#label-max-data-extension-time-in-days). (see [below for nested schema](#nestedblock--max_data_extension_time_in_days))
- `replace_invalid_characters` (Boolean) Specifies whether to replace invalid UTF-8 characters with the Unicode replacement character (�) in query results for an Iceberg table. You can only set this parameter for tables that use an external Iceberg catalog.
- `storage_serialization_policy` (String) Specifies the storage serialization policy for Iceberg tables that use Snowflake as the catalog. Valid options are: [COMPATIBLE OPTIMIZED]. COMPATIBLE: Snowflake performs encoding and compression of data files that ensures interoperability with third-party compute engines. OPTIMIZED: Snowflake performs encoding and compression of data files that ensures the best table performance within Snowflake.
- `trace_level` (String) Controls how trace events are ingested into the event table. Valid options are: [ALWAYS ON_EVENT OFF]. For information about levels, see [TRACE_LEVEL](https://docs.snowflake.com/en/sql-reference/parameters.html#label-trace-level).

### Read-Only

- `id` (String) The ID of this resource.

<a id="nestedblock--data_retention_time_in_days"></a>
### Nested Schema for `data_retention_time_in_days`

Required:

- `value` (Number)


<a id="nestedblock--max_data_extension_time_in_days"></a>
### Nested Schema for `max_data_extension_time_in_days`

Required:

- `value` (Number)

## Import

Import is supported using the following syntax:

```shell
terraform import snowflake_secondary_database.example 'secondary_database_name'
```
79 changes: 79 additions & 0 deletions docs/resources/shared_database.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
page_title: "snowflake_shared_database Resource - terraform-provider-snowflake"
subcategory: ""
description: |-
A shared database creates a database from a share provided by another Snowflake account. For more information about shares, see Introduction to Secure Data Sharing https://docs.snowflake.com/en/user-guide/data-sharing-intro.
---

# snowflake_shared_database (Resource)

A shared database creates a database from a share provided by another Snowflake account. For more information about shares, see [Introduction to Secure Data Sharing](https://docs.snowflake.com/en/user-guide/data-sharing-intro).

## Example Usage

```terraform
# 1. Preparing database to share
resource "snowflake_share" "test" {
provider = primary_account # notice the provider fields
name = "share_name"
accounts = ["<secondary_account_organization_name>.<secondary_account_name>"]
}

resource "snowflake_database" "test" {
provider = primary_account
name = "shared_database"
}

resource "snowflake_grant_privileges_to_share" "test" {
provider = primary_account
to_share = snowflake_share.test.name
privileges = ["USAGE"]
on_database = snowflake_database.test.name
}

# 2. Creating shared database
resource "snowflake_shared_database" "test" {
sfc-gh-asawicki marked this conversation as resolved.
Show resolved Hide resolved
provider = secondary_account
depends_on = [snowflake_grant_privileges_to_share.test]
name = snowflake_database.test.name # shared database should have the same as the "imported" one
from_share = "<primary_account_organization_name>.<primary_account_name>.${snowflake_share.test.name}"
is_transient = false
external_volume = "external_volume_name"
catalog = "catalog_name"
default_ddl_collation = "en_US"
log_level = "OFF"
trace_level = "OFF"
comment = "A shared database"
}
```

<!-- schema generated by tfplugindocs -->
## Schema

### Required

- `from_share` (String) A fully qualified path to a share from which the database will be created. A fully qualified path follows the format of `"<provider_account>"."<share_name>"`.
- `name` (String) Specifies the identifier for the database; must be unique for your account.

### Optional

- `catalog` (String) The database parameter that specifies the default catalog to use for Iceberg tables.
- `comment` (String) Specifies a comment for the database.
- `default_ddl_collation` (String) Specifies a default collation specification for all schemas and tables added to the database. It can be overridden on schema or table level. For more information, see [collation specification](https://docs.snowflake.com/en/sql-reference/collation#label-collation-specification).
- `external_volume` (String) The database parameter that specifies the default external volume to use for Iceberg tables.
- `log_level` (String) Specifies the severity level of messages that should be ingested and made available in the active event table. Valid options are: [TRACE DEBUG INFO WARN ERROR FATAL OFF]. Messages at the specified level (and at more severe levels) are ingested. For more information, see [LOG_LEVEL](https://docs.snowflake.com/en/sql-reference/parameters.html#label-log-level).
- `replace_invalid_characters` (Boolean) Specifies whether to replace invalid UTF-8 characters with the Unicode replacement character (�) in query results for an Iceberg table. You can only set this parameter for tables that use an external Iceberg catalog.
- `storage_serialization_policy` (String) Specifies the storage serialization policy for Iceberg tables that use Snowflake as the catalog. Valid options are: [COMPATIBLE OPTIMIZED]. COMPATIBLE: Snowflake performs encoding and compression of data files that ensures interoperability with third-party compute engines. OPTIMIZED: Snowflake performs encoding and compression of data files that ensures the best table performance within Snowflake.
- `trace_level` (String) Controls how trace events are ingested into the event table. Valid options are: [ALWAYS ON_EVENT OFF]. For information about levels, see [TRACE_LEVEL](https://docs.snowflake.com/en/sql-reference/parameters.html#label-trace-level).

### Read-Only

- `id` (String) The ID of this resource.

## Import

Import is supported using the following syntax:

```shell
terraform import snowflake_shared_database.example 'shared_database_name'
```
1 change: 1 addition & 0 deletions examples/resources/snowflake_secondary_database/import.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
terraform import snowflake_secondary_database.example 'secondary_database_name'
32 changes: 32 additions & 0 deletions examples/resources/snowflake_secondary_database/resource.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# 1. Preparing primary database
resource "snowflake_database" "primary" {
provider = primary_account # notice the provider fields
name = "database_name"
replication_configuration {
accounts = ["<secondary_account_organization_name>.<secondary_account_name>"]
ignore_edition_check = true
}
}

# 2. Creating secondary database
resource "snowflake_secondary_database" "test" {
provider = secondary_account
name = snowflake_database.primary.name # It's recommended to give a secondary database the same name as its primary database
as_replica_of = "<primary_account_organization_name>.<primary_account_name>.${snowflake_database.primary.name}"
is_transient = false

data_retention_time_in_days {
value = 10
}

max_data_extension_time_in_days {
value = 20
}

external_volume = "external_volume_name"
catalog = "catalog_name"
default_ddl_collation = "en_US"
log_level = "OFF"
trace_level = "OFF"
comment = "A secondary database"
}
1 change: 1 addition & 0 deletions examples/resources/snowflake_shared_database/import.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
terraform import snowflake_shared_database.example 'shared_database_name'
33 changes: 33 additions & 0 deletions examples/resources/snowflake_shared_database/resource.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# 1. Preparing database to share
resource "snowflake_share" "test" {
provider = primary_account # notice the provider fields
name = "share_name"
accounts = ["<secondary_account_organization_name>.<secondary_account_name>"]
}

resource "snowflake_database" "test" {
provider = primary_account
name = "shared_database"
}

resource "snowflake_grant_privileges_to_share" "test" {
provider = primary_account
to_share = snowflake_share.test.name
privileges = ["USAGE"]
on_database = snowflake_database.test.name
}

# 2. Creating shared database
resource "snowflake_shared_database" "test" {
provider = secondary_account
depends_on = [snowflake_grant_privileges_to_share.test]
name = snowflake_database.test.name # shared database should have the same as the "imported" one
from_share = "<primary_account_organization_name>.<primary_account_name>.${snowflake_share.test.name}"
is_transient = false
external_volume = "external_volume_name"
catalog = "catalog_name"
default_ddl_collation = "en_US"
log_level = "OFF"
trace_level = "OFF"
comment = "A shared database"
}
6 changes: 6 additions & 0 deletions pkg/acceptance/check_destroy.go
Original file line number Diff line number Diff line change
Expand Up @@ -136,12 +136,18 @@ var showByIdFunctions = map[resources.Resource]showByIdFunc{
resources.Schema: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.Schemas.ShowByID)
},
resources.SecondaryDatabase: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.Databases.ShowByID)
},
resources.Sequence: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.Sequences.ShowByID)
},
resources.Share: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.Shares.ShowByID)
},
resources.SharedDatabase: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.Databases.ShowByID)
},
resources.Stage: func(ctx context.Context, client *sdk.Client, id sdk.ObjectIdentifier) error {
return runShowById(ctx, id, client.Stages.ShowByID)
},
Expand Down
24 changes: 24 additions & 0 deletions pkg/acceptance/helpers/database_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,30 @@ func (c *DatabaseClient) client() sdk.Databases {
return c.context.client.Databases
}

func (c *DatabaseClient) CreatePrimaryDatabase(t *testing.T, enableReplicationTo []sdk.AccountIdentifier) (*sdk.Database, sdk.ExternalObjectIdentifier, func()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit-pick: let's not make it the first method in this file (look where secondary is located)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it under secondary.

t.Helper()
ctx := context.Background()

primaryDatabase, primaryDatabaseCleanup := c.CreateDatabase(t)

err := c.client().AlterReplication(ctx, primaryDatabase.ID(), &sdk.AlterDatabaseReplicationOptions{
EnableReplication: &sdk.EnableReplication{
ToAccounts: enableReplicationTo,
IgnoreEditionCheck: sdk.Bool(true),
},
})
require.NoError(t, err)

organizationName, err := c.context.client.ContextFunctions.CurrentOrganizationName(ctx)
require.NoError(t, err)

accountName, err := c.context.client.ContextFunctions.CurrentAccountName(ctx)
require.NoError(t, err)

externalPrimaryId := sdk.NewExternalObjectIdentifier(sdk.NewAccountIdentifier(organizationName, accountName), primaryDatabase.ID())
return primaryDatabase, externalPrimaryId, primaryDatabaseCleanup
}

func (c *DatabaseClient) CreateDatabase(t *testing.T) (*sdk.Database, func()) {
t.Helper()
return c.CreateDatabaseWithOptions(t, c.ids.RandomAccountObjectIdentifier(), &sdk.CreateDatabaseOptions{})
Expand Down
2 changes: 2 additions & 0 deletions pkg/provider/provider.go
Original file line number Diff line number Diff line change
Expand Up @@ -494,9 +494,11 @@ func getResources() map[string]*schema.Resource {
"snowflake_saml_integration": resources.SAMLIntegration(),
"snowflake_schema": resources.Schema(),
"snowflake_scim_integration": resources.SCIMIntegration(),
"snowflake_secondary_database": resources.SecondaryDatabase(),
"snowflake_sequence": resources.Sequence(),
"snowflake_session_parameter": resources.SessionParameter(),
"snowflake_share": resources.Share(),
"snowflake_shared_database": resources.SharedDatabase(),
"snowflake_stage": resources.Stage(),
"snowflake_storage_integration": resources.StorageIntegration(),
"snowflake_stream": resources.Stream(),
Expand Down
2 changes: 2 additions & 0 deletions pkg/provider/resources/resources.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,10 @@ const (
Role resource = "snowflake_role"
RowAccessPolicy resource = "snowflake_row_access_policy"
Schema resource = "snowflake_schema"
SecondaryDatabase resource = "snowflake_secondary_database"
Sequence resource = "snowflake_sequence"
Share resource = "snowflake_share"
SharedDatabase resource = "snowflake_shared_database"
Stage resource = "snowflake_stage"
StorageIntegration resource = "snowflake_storage_integration"
Stream resource = "snowflake_stream"
Expand Down
Loading
Loading