terraform-aws-snowflake-loader-setup

A Terraform module for preparing snowflake for loading snowplow data. Should be used after the terraform-snowflake-target.

Prerequisites

Authentication for the service user is required for the snowflake terraform provider - follow this tutorial to obtain snowflake connection details:

Parameter	Description
account	The account name.
username	A snowflake user to perform resource creation.
region	Region for the snowflake deployment.
role	Needs to be ACCOUNTADMIN or similar.
private_key_path	Path the private key.

Usage

Applying module directly

1. Fill variables in terraform.tfvars.tmpl and copy it to terraform.tfvars.
2. Using environment variables for authentication as described in here. Fill the template in snowflake_provider_vars.sh and run source ./snowflake_provider_vars.sh to set up your local environment.
3. Run terraform init
4. Run terraform apply

Using the module from another module

Pass authentication parameters directly to Snowflake provider.

provider "snowflake" {
  username         = "my_user"
  account          = "my_account"
  region           = "us-west-2"
  role             = "ACCOUNTADMIN"
  private_key_path = "/path/to/private/key"
}

module "snowflake_target" {
   source = "snowplow-devops/target/snowflake"
   
   name               = "snowplow"
   snowflake_password = "example_password"
}

module "snowflake_setup" {
   source = "snowplow-devops/snowflake-loader-setup/aws"

   name                  = "snowplow"
   stage_bucket          = "snowplow-oss-bucket"
   aws_account_id        = "0000000000"
   snowflake_database    = module.snowflake_target.snowflake_database
   snowflake_event_table = module.snowflake_target.snowflake_event_table
   snowflake_file_format = module.snowflake_target.snowflake_file_format
   snowflake_schema      = module.snowflake_target.snowflake_schema
   snowflake_user        = module.snowflake_target.snowflake_user
}

Requirements

Name	Version
terraform	>= 1.0.0
snowflake	0.45.0

Providers

Name	Version
snowflake	0.45.0

Modules

No modules.

Resources

Name	Type
snowflake_database_grant.loader	resource
snowflake_file_format_grant.loader	resource
snowflake_integration_grant.loader	resource
snowflake_role.loader	resource
snowflake_role_grants.loader	resource
snowflake_schema_grant.loader	resource
snowflake_stage.folder_monitoring	resource
snowflake_stage.transformed	resource
snowflake_stage_grant.folder_monitoring	resource
snowflake_stage_grant.transformed	resource
snowflake_storage_integration.integration	resource
snowflake_table_grant.loader	resource
snowflake_warehouse.loader	resource
snowflake_warehouse_grant.loader	resource

Inputs

Name	Description	Type	Default	Required
aws_account_id	AWS account id for the role to be linked with storage integration. Role itself won't be created by this module	string	n/a	yes
name	A name which will be prepended to the created resources	string	n/a	yes
snowflake_database	Snowflake database name	string	n/a	yes
snowflake_event_table	Snowflake events table name	string	n/a	yes
snowflake_file_format	Snowflake file format name	string	n/a	yes
snowflake_schema	Snowflake schema name	string	n/a	yes
snowflake_user	Snowflake username	string	n/a	yes
stage_bucket	Name of the S3 bucket which will be used as stage by Snowflake	string	n/a	yes
folder_monitoring_enabled	Folder monitoring loading	bool	false	no
override_folder_monitoring_stage_url	Override monitoring stage url, if not set it will be defaulted "s3://${var.stage_bucket_name}/${var.name}/shredded/v1"	string	""	no
override_iam_loader_role_name	Override integration iam role name, if not set it will be var.name with -snowflakedb-load-role suffix	string	""	no
override_snowflake_loader_role	Override loader role name in snowflake, if not set it will be uppercase var.name with "_LOADER_ROLE" suffix	string	""	no
override_snowflake_loader_user	Override loader user name in snowflake, if not set it will be uppercase var.name with _LOADER_USER suffix	string	""	no
override_snowflake_wh_name	Override warehouse name, if not set it will be defaulted to uppercase var.name with "_WAREHOUSE" suffix	string	""	no
override_transformed_stage_url	Override transformed stage url, if not set it will be defaulted "s3://${var.stage_bucket_name}/${var.name}/monitoring"	string	""	no
snowflake_wh_auto_resume	Whether to enable auto resume which makes automatically resume the warehouse when any statement that requires a warehouse is submitted	bool	true	no
snowflake_wh_auto_suspend	Time period to wait before suspending warehouse	number	60	no
snowflake_wh_size	Size of the Snowflake warehouse to connect to	string	"XSMALL"	no

Outputs

Name	Description
aws_iam_storage_integration_role_arn	AWS storage integration IAM role ARN
aws_iam_storage_integration_role_name	AWS storage integration IAM role name
aws_iam_storage_integration_user_arn	AWS storage integration IAM user ARN
aws_s3_bucket_name	Name of the S3 bucket which will be used as stage by Snowflake
aws_s3_folder_monitoring_stage_url	AWS bucket url of folder monitoring stage
aws_s3_transformed_stage_url	AWS bucket url of transformed stage
aws_storage_external_id	AWS external id
snowflake_loader_role	Snowflake role for loading snowplow data
snowflake_monitoring_stage_name	Name of monitoring stage
snowflake_transformed_stage_name	Name of transformed stage
snowflake_warehouse	Snowflake warehouse name

Copyright and license

The Terraform Snowflake Loader Setup project is Copyright 2022-2022 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.