Terraform Athena module to create new modules using this as baseline
We eat, drink, sleep and most importantly love DevOps. We are working towards strategies for standardizing architecture while ensuring security for the infrastructure. We are strong believer of the philosophy Bigger problems are always solved by breaking them into smaller manageable problems. Resonating with microservices architecture, it is considered best-practice to run database, cluster, storage in smaller connected yet manageable pieces within the infrastructure.
This module is basically combination of Terraform open source and includes automatation tests and examples. It also helps to create and improve your infrastructure with minimalistic code instead of maintaining the whole infrastructure code yourself.
We have fifty plus terraform modules. A few of them are comepleted and are available for open source usage while a few others are in progress.
This module has a few dependencies:
IMPORTANT: Since the master
branch used in source
varies based on new modifications, we suggest that you use the release versions here.
Here are some examples of how you can use this module in your inventory structure:
module "athena" {
source = "clouddrove/athena/aws"
version = "1.0.0"
name = "athena"
environment = "test"
label_order = ["name", "environment"]
enabled = true
workgroup_force_destroy = true
# S3 Bucket Configuration
bucket_force_destroy = true
s3_output_path = "accessLogs/queryresults/" # The S3 bucket path used to store query results
# Database for Athena
databases = {
database1 = {
force_destroy = true
properties = {
custom_prop_1 = "example"
}
}
}
}
locals {
name = "athena"
environment = "test"
label_order = ["name", "environment"]
}
module "s3_bucket" {
source = "clouddrove/s3/aws"
version = "1.3.0"
name = format("%s-bucket-test", local.name)
versioning = true
acl = "private"
force_destroy = true
}
module "athena" {
source = "clouddrove/athena/aws"
version = "1.0.0"
name = local.name
environment = local.environment
label_order = local.label_order
enabled = true
workgroup_force_destroy = true
# S3 Bucket Configuration
create_s3_bucket = false
athena_s3_bucket_id = module.s3_bucket.id
s3_output_path = "outputs/" # The S3 bucket path used to store query results
# Database for Athena
databases = {
database1 = {
force_destroy = true
properties = {
custom_prop_1 = "example"
}
}
}
# Data catalog to test terraform
data_catalogs = {
glue1 = {
description = "This is an example to test Terraform"
type = "GLUE"
parameters = {
catalog-id : "123456789012" # The catalog_id is the account ID of the AWS account to which the AWS Glue catalog belongs.
}
}
}
# Named Queries to test terarform
named_queries = {
query1 = {
database = "database1"
description = "This is an example query to test Terraform"
query = "SELECT * FROM %s limit 10;"
}
}
}
Name | Description | Type | Default | Required |
---|---|---|---|---|
athena_kms_key | Use an existing KMS key for Athena if create_workgroup_kms_key is false . |
string |
null |
no |
athena_s3_bucket_id | Use an existing S3 bucket for Athena query results if create_s3_bucket is false . |
string |
null |
no |
bucket_acl | Canned ACL to apply to the S3 bucket. | string |
null |
no |
bucket_force_destroy | A boolean that indicates all objects should be deleted from the bucket so that the bucket can be destroyed without error. These objects are not recoverable. | bool |
false |
no |
bucket_label_order | Label order, e.g. name ,application for S3 Bucket. |
list(any) |
[ |
no |
bucket_versioning | Enable Versioning of S3. | bool |
true |
no |
bytes_scanned_cutoff_per_query | Integer for the upper data usage limit (cutoff) for the amount of bytes a single query in a workgroup is allowed to scan. Must be at least 10485760. | number |
null |
no |
create_database_kms_key | Enable the creation of a KMS key used by Athena database. | bool |
true |
no |
create_s3_bucket | Conditionally create S3 bucket. | bool |
true |
no |
create_workgroup_kms_key | Enable the creation of a KMS key used by Athena workgroup. | bool |
true |
no |
data_catalogs | Map of Athena data catalogs and related configuration. | map(any) |
{} |
no |
databases | Map of Athena databases and related configuration. | map(any) |
n/a | yes |
deletion_window_in_days | Duration in days after which the key is deleted after destruction of the resource. | number |
7 |
no |
enabled | Set to false to prevent the module from creating AWS Athena related resources. | bool |
false |
no |
enforce_workgroup_configuration | Boolean whether the settings for the workgroup override client-side settings. | bool |
true |
no |
environment | Environment (e.g. prod , dev , staging ). |
string |
"" |
no |
kms_key_enabled | Specifies whether the kms is enabled or disabled. | bool |
true |
no |
label_order | Label order, e.g. name ,application . |
list(any) |
[] |
no |
managedby | ManagedBy, eg 'CloudDrove'. | string |
"hello@clouddrove.com" |
no |
multi_region | Indicates whether the KMS key is a multi-Region (true) or regional (false) key. | bool |
true |
no |
name | Name (e.g. app or cluster ). |
string |
"" |
no |
named_queries | Map of Athena named queries and related configuration. | map(map(string)) |
{} |
no |
publish_cloudwatch_metrics_enabled | Boolean whether Amazon CloudWatch metrics are enabled for the workgroup. | bool |
true |
no |
repository | Terraform current module repo | string |
"https://github.com/clouddrove/terraform-aws-athena" |
no |
s3_output_path | The S3 bucket path used to store query results. | string |
"" |
no |
tags | Additional tags (e.g. map(BusinessUnit ,XYZ ). |
map(any) |
{} |
no |
workgroup_encryption_option | Indicates whether Amazon S3 server-side encryption with Amazon S3-managed keys (SSE_S3), server-side encryption with KMS-managed keys (SSE_KMS), or client-side encryption with KMS-managed keys (CSE_KMS) is used. | string |
"SSE_KMS" |
no |
workgroup_force_destroy | The option to delete the workgroup and its contents even if the workgroup contains any named queries. | bool |
false |
no |
Name | Description |
---|---|
bucket_arn | ID of S3 bucket used by Athena. |
bucket_id | ID of S3 bucket used by Athena. |
data_catalogs | List of newly created Athena data catalogs. |
databases | List of newly created Athena databases. |
kms_key_arn | ARN of KMS key used by Athena. |
named_queries | List of newly created Athena named queries. |
workgroup_id | ID of newly created Athena workgroup. |
In this module testing is performed with terratest and it creates a small piece of infrastructure, matches the output like ARN, ID and Tags name etc and destroy infrastructure in your AWS account. This testing is written in GO, so you need a GO environment in your system.
You need to run the following command in the testing folder:
go test -run Test
If you come accross a bug or have any feedback, please log it in our issue tracker, or feel free to drop us an email at hello@clouddrove.com.
If you have found it worth your time, go ahead and give us a ★ on our GitHub!
At CloudDrove, we offer expert guidance, implementation support and services to help organisations accelerate their journey to the cloud. Our services include docker and container orchestration, cloud migration and adoption, infrastructure automation, application modernisation and remediation, and performance engineering.
We are The Cloud Experts!
We ❤️ Open Source and you can check out our other modules to get help with your new Cloud ideas.